From C.Klaij at marin.nl Fri Jun 1 03:36:12 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 1 Jun 2012 08:36:12 +0000 Subject: [petsc-users] GMRES pc right fails while pc left succeeds Message-ID: To solve the incompressible Navier-Stokes equations, I'm using cell-centered finite volumes and Picard linearization. At each non-linear iteration, I'm using one linear (Krylov) iteration with a SIMPLE-type block preconditioner (for example Elman e.a, JCP 227, 2008). The matrix and preconditioner are shells. Below, I'm showing the results for the first few non-linear iterations using (1) KSPRICHARDSON to check that the shells are ok (2) GMRES with right preconditioning (3) GMRES with left preconditioning To my surprise (2) fails while (1) and (3) succeed. The reason I'm interested in right preconditioning is that SIMPLE is a variable preconditioner so in theory I should be using FGMRES as soon as I increase the number of linear (Krylov) iterations. What could be the reason that GMRES fails with right pc? KSPRICHARDSON =========================================================================== KSP Object:(sys_) 64 MPI processes type: richardson Richardson: damping factor=1 maximum iterations=1, initial guess is zero tolerances: relative=0.1, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object:(sys_) 64 MPI processes type: shell Shell: no name linear system matrix = precond matrix: Matrix Object: 64 MPI processes type: shell rows=13599580, cols=13599580 Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.002292124139e+07 true resid norm 2.449788038787e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.466728998567e+08 true resid norm 1.184153811354e+02 ||r(i)||/||b|| 4.833699049085e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.633884476245e+06 true resid norm 2.178463431810e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.026742626974e+07 true resid norm 7.583593359073e+01 ||r(i)||/||b|| 3.481166242379e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.603912564752e+06 true resid norm 2.063977595102e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.334118865425e+07 true resid norm 4.644866011115e+01 ||r(i)||/||b|| 2.250444007792e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.837763400164e+06 true resid norm 1.744796447506e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.366091965735e+07 true resid norm 3.667361782612e+01 ||r(i)||/||b|| 2.101885172826e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.164837635443e+06 true resid norm 1.379343201904e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.076321593136e+07 true resid norm 2.580128984813e+01 ||r(i)||/||b|| 1.870548954932e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.740236061285e+06 true resid norm 1.089190060245e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.774062456490e+07 true resid norm 1.905053703637e+01 ||r(i)||/||b|| 1.749055351467e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.116828431953e+06 true resid norm 8.739520919331e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.292581773188e+07 true resid norm 1.417058997539e+01 ||r(i)||/||b|| 1.621437846101e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.732540786832e+06 true resid norm 7.103614423801e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.024675862117e+07 true resid norm 1.143160619326e+01 ||r(i)||/||b|| 1.609266144141e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.352305197787e+06 true resid norm 5.785689719087e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.564758442669e+06 true resid norm 8.227117972441e+00 ||r(i)||/||b|| 1.421977045416e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.150715386196e+06 true resid norm 4.793529363445e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.372682588351e+06 true resid norm 6.688883029785e+00 ||r(i)||/||b|| 1.395398363635e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.572132583239e+05 true resid norm 3.988785666972e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.261708756365e+06 true resid norm 6.009243780630e+00 ||r(i)||/||b|| 1.506534640452e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.708272340553e+05 true resid norm 3.328712835238e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.063709557864e+06 true resid norm 4.716564750520e+00 ||r(i)||/||b|| 1.416933506727e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.237006694662e+05 true resid norm 2.840137112965e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.236149418914e+06 true resid norm 3.966057508670e+00 ||r(i)||/||b|| 1.396431704148e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.836468734124e+05 true resid norm 2.463325405601e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.222197400481e+06 true resid norm 3.320264231000e+00 ||r(i)||/||b|| 1.347878856545e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.369688632472e+05 true resid norm 2.149175735231e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.330816379175e+06 true resid norm 2.741356483857e+00 ||r(i)||/||b|| 1.275538542018e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.363278393552e+05 true resid norm 1.909837709921e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.681890437397e+06 true resid norm 2.308561048035e+00 ||r(i)||/||b|| 1.208773413596e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.802377915000e+05 true resid norm 1.728472606236e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.336982911437e+06 true resid norm 1.968373614139e+00 ||r(i)||/||b|| 1.138793641876e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.365927099152e+05 true resid norm 1.596065109380e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.049884061323e+06 true resid norm 1.732502774165e+00 ||r(i)||/||b|| 1.085483771297e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.002859406874e+05 true resid norm 1.495493921635e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.497026355879e+05 true resid norm 1.561301201038e+00 ||r(i)||/||b|| 1.044003709043e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.860953637588e+05 true resid norm 1.414036045246e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.970043961017e+05 true resid norm 1.414991682369e+00 ||r(i)||/||b|| 1.000675822321e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.644104206835e+05 true resid norm 1.346916532259e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.648634021008e+05 true resid norm 1.246596455132e+00 ||r(i)||/||b|| 9.255187127602e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.872174371851e+05 true resid norm 1.290333542891e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.372959177536e+05 true resid norm 1.202317588926e+00 ||r(i)||/||b|| 9.317882151865e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.661343842757e+05 true resid norm 1.243144485951e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.563429909910e+05 true resid norm 1.057154297198e+00 ||r(i)||/||b|| 8.503873114867e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.672029389906e+05 true resid norm 1.199158775660e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.625434467940e+05 true resid norm 1.018232434251e+00 ||r(i)||/||b|| 8.491222804841e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.609863909906e+05 true resid norm 1.166442633617e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.594136262967e+05 true resid norm 8.852250142544e-01 ||r(i)||/||b|| 7.589100301566e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.589084188509e+05 true resid norm 1.133575353542e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.531640641766e+05 true resid norm 8.680191349580e-01 ||r(i)||/||b|| 7.657357159767e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.545741532506e+05 true resid norm 1.109246759396e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.068835202912e+05 true resid norm 7.541905012706e-01 ||r(i)||/||b|| 6.799122872184e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.505271454290e+05 true resid norm 1.083901826677e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.895666749586e+05 true resid norm 7.334526058239e-01 ||r(i)||/||b|| 6.766780789296e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.429767942414e+05 true resid norm 1.065392345898e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.674722288653e+05 true resid norm 6.605153582236e-01 ||r(i)||/||b|| 6.199738159999e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.349552981353e+05 true resid norm 1.046479546594e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.420791854107e+05 true resid norm 6.061030158236e-01 ||r(i)||/||b|| 5.791828591358e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.294549761112e+05 true resid norm 1.030516295057e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.009630735908e+05 true resid norm 5.682902043831e-01 ||r(i)||/||b|| 5.514616383159e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.263203266274e+05 true resid norm 1.016513956599e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.685817059997e+05 true resid norm 5.312671230577e-01 ||r(i)||/||b|| 5.226363293968e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.218894584644e+05 true resid norm 1.004638417180e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.506946781986e+05 true resid norm 5.019080583246e-01 ||r(i)||/||b|| 4.995907480160e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.140914318170e+05 true resid norm 9.947019980499e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.382103111998e+05 true resid norm 4.655872827257e-01 ||r(i)||/||b|| 4.680671031510e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.067119548244e+05 true resid norm 9.863770120848e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.108428061151e+05 true resid norm 4.424022035790e-01 ||r(i)||/||b|| 4.485122809624e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.019474184396e+05 true resid norm 9.799542793429e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.721773506635e+05 true resid norm 4.199861021046e-01 ||r(i)||/||b|| 4.285772417732e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.830423129037e+04 true resid norm 9.751559280594e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.361650301572e+05 true resid norm 3.923976141692e-01 ||r(i)||/||b|| 4.023947379883e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.330148868500e+04 true resid norm 9.719876905233e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.197457401960e+05 true resid norm 3.711703982110e-01 ||r(i)||/||b|| 3.818673855954e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.685377015665e+04 true resid norm 9.705157913387e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.050538464657e+05 true resid norm 3.521450162021e-01 ||r(i)||/||b|| 3.628431596320e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.104143121935e+04 true resid norm 9.707989920233e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.788424265633e+05 true resid norm 3.340342892670e-01 ||r(i)||/||b|| 3.440818253950e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.724089365112e+04 true resid norm 9.730581336050e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.457900621193e+05 true resid norm 3.145704907970e-01 ||r(i)||/||b|| 3.232802644911e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.429452857765e+04 true resid norm 9.778147647982e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.229330560666e+05 true resid norm 3.032903181340e-01 ||r(i)||/||b|| 3.101715468538e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.995091708215e+04 true resid norm 9.844124677010e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.109695743465e+05 true resid norm 2.925574699740e-01 ||r(i)||/||b|| 2.971899275689e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.431621141247e+04 true resid norm 9.932717704670e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.988547543300e+05 true resid norm 2.828386955713e-01 ||r(i)||/||b|| 2.847545898122e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.059506259530e+04 true resid norm 1.004501645405e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.817009034402e+05 true resid norm 2.805123677650e-01 ||r(i)||/||b|| 2.792552596088e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.745261235425e+04 true resid norm 1.017758014319e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.542169446329e+05 true resid norm 2.724197244073e-01 ||r(i)||/||b|| 2.676664988874e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.516674377618e+04 true resid norm 1.034201742913e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.378530798956e+05 true resid norm 2.667643670507e-01 ||r(i)||/||b|| 2.579422911232e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.194287548926e+04 true resid norm 1.053156823791e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.290402092140e+05 true resid norm 2.646974039255e-01 ||r(i)||/||b|| 2.513371208787e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.808857001240e+04 true resid norm 1.074849466996e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.215939060099e+05 true resid norm 2.616032993026e-01 ||r(i)||/||b|| 2.433859878386e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.475831147279e+04 true resid norm 1.099007926051e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.083656356962e+05 true resid norm 2.560599441241e-01 ||r(i)||/||b|| 2.329918993799e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.221563346784e+04 true resid norm 1.125751964901e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.881930780060e+05 true resid norm 2.531356206610e-01 ||r(i)||/||b|| 2.248591417590e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.150036386185e+04 true resid norm 1.154770933556e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.799205358655e+05 true resid norm 2.606336712952e-01 ||r(i)||/||b|| 2.257016207470e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.926677498970e+04 true resid norm 1.185826544114e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.729849090921e+05 true resid norm 2.633851055090e-01 ||r(i)||/||b|| 2.221109881679e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.669501750810e+04 true resid norm 1.218539067690e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.693046383608e+05 true resid norm 2.628742815537e-01 ||r(i)||/||b|| 2.157290550003e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.426522258130e+04 true resid norm 1.253394141865e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.611167773181e+05 true resid norm 2.560938856149e-01 ||r(i)||/||b|| 2.043203147845e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.265287560670e+04 true resid norm 1.289828098593e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.491234168952e+05 true resid norm 2.613672993010e-01 ||r(i)||/||b|| 2.026373123567e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.197888583685e+04 true resid norm 1.327073787929e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.443230044262e+05 true resid norm 2.706087663843e-01 ||r(i)||/||b|| 2.039138809354e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.073727378270e+04 true resid norm 1.365403569274e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.402005787526e+05 true resid norm 2.764196786979e-01 ||r(i)||/||b|| 2.024454050936e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.932459486969e+04 true resid norm 1.403841601895e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.387678749811e+05 true resid norm 2.808710347819e-01 ||r(i)||/||b|| 2.000731666612e-01 KSPGMRES, PC_RIGHT =========================================================================== KSP Object:(sys_) 64 MPI processes type: gmres GMRES: restart=30, using Modified Gram-Schmidt Orthogonalization GMRES: happy breakdown tolerance 1e-30 maximum iterations=1, initial guess is zero tolerances: relative=0.1, absolute=1e-50, divergence=10000 right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object:(sys_) 64 MPI processes type: shell Shell: no name linear system matrix = precond matrix: Matrix Object: 64 MPI processes type: shell rows=13599580, cols=13599580 Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.449788038787e+01 true resid norm 2.449788038787e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.398039251803e+01 true resid norm 2.398039251803e+01 ||r(i)||/||b|| 9.788762186097e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.420226272871e+01 true resid norm 2.420226272871e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.377979701744e+01 true resid norm 2.377979701744e+01 ||r(i)||/||b|| 9.825443713255e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.390757858574e+01 true resid norm 2.390757858574e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.357584168912e+01 true resid norm 2.357584168912e+01 ||r(i)||/||b|| 9.861241950775e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.365589714406e+01 true resid norm 2.365589714406e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.338729986835e+01 true resid norm 2.338729986835e+01 ||r(i)||/||b|| 9.886456525376e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.343424425490e+01 true resid norm 2.343424425490e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.321506908653e+01 true resid norm 2.321506908653e+01 ||r(i)||/||b|| 9.906472269394e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.323743871191e+01 true resid norm 2.323743871191e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.306701649761e+01 true resid norm 2.306701649761e+01 ||r(i)||/||b|| 9.926660499717e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.306939620566e+01 true resid norm 2.306939620566e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.293212356739e+01 true resid norm 2.293212356739e+01 ||r(i)||/||b|| 9.940495781924e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.292139442317e+01 true resid norm 2.292139442317e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.280900557416e+01 true resid norm 2.280900557416e+01 ||r(i)||/||b|| 9.950967708624e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.278915281573e+01 true resid norm 2.278915281573e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.269700749818e+01 true resid norm 2.269700749818e+01 ||r(i)||/||b|| 9.959566150487e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.267076491906e+01 true resid norm 2.267076491906e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.259790463024e+01 true resid norm 2.259790463024e+01 ||r(i)||/||b|| 9.967861565731e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.256763385923e+01 true resid norm 2.256763385923e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.250777172301e+01 true resid norm 2.250777172301e+01 ||r(i)||/||b|| 9.973474340910e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.247501724325e+01 true resid norm 2.247501724325e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.242579138506e+01 true resid norm 2.242579138506e+01 ||r(i)||/||b|| 9.978097521503e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.239174013732e+01 true resid norm 2.239174013732e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.235122856474e+01 true resid norm 2.235122856474e+01 ||r(i)||/||b|| 9.981907805138e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.231677642916e+01 true resid norm 2.231677642916e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.228341200759e+01 true resid norm 2.228341200759e+01 ||r(i)||/||b|| 9.985049623239e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.224922495513e+01 true resid norm 2.224922495513e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.222172256133e+01 true resid norm 2.222172256133e+01 ||r(i)||/||b|| 9.987638942997e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.218828968686e+01 true resid norm 2.218828968686e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.216560438029e+01 true resid norm 2.216560438029e+01 ||r(i)||/||b|| 9.989776000364e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.213327413534e+01 true resid norm 2.213327413534e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.211455047562e+01 true resid norm 2.211455047562e+01 ||r(i)||/||b|| 9.991540492564e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.208356267974e+01 true resid norm 2.208356267974e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.206809964045e+01 true resid norm 2.206809964045e+01 ||r(i)||/||b|| 9.992997941721e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.203860985605e+01 true resid norm 2.203860985605e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.202583242681e+01 true resid norm 2.202583242681e+01 ||r(i)||/||b|| 9.994202252626e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.199793147806e+01 true resid norm 2.199793147806e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.198736759090e+01 true resid norm 2.198736759090e+01 ||r(i)||/||b|| 9.995197781584e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.196109697446e+01 true resid norm 2.196109697446e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.195235874753e+01 true resid norm 2.195235874753e+01 ||r(i)||/||b|| 9.996021042602e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.192772281273e+01 true resid norm 2.192772281273e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.192049128260e+01 true resid norm 2.192049128260e+01 ||r(i)||/||b|| 9.996702106190e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.189746681274e+01 true resid norm 2.189746681274e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.189218569645e+01 true resid norm 2.189218569645e+01 ||r(i)||/||b|| 9.997588252409e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.187196087205e+01 true resid norm 2.187196087205e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.186758679430e+01 true resid norm 2.186758679430e+01 ||r(i)||/||b|| 9.998000143755e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.184881028281e+01 true resid norm 2.184881028281e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.184355960145e+01 true resid norm 2.184355960145e+01 ||r(i)||/||b|| 9.997596811319e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.182297473777e+01 true resid norm 2.182297473777e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.181859649271e+01 true resid norm 2.181859649271e+01 ||r(i)||/||b|| 9.997993745070e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.179944311209e+01 true resid norm 2.179944311209e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.179579052478e+01 true resid norm 2.179579052478e+01 ||r(i)||/||b|| 9.998324458429e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.177799695396e+01 true resid norm 2.177799695396e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.177494837915e+01 true resid norm 2.177494837915e+01 ||r(i)||/||b|| 9.998600158305e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.175844331512e+01 true resid norm 2.175844331512e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.175631168374e+01 true resid norm 2.175631168374e+01 ||r(i)||/||b|| 9.999020319905e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.174229725863e+01 true resid norm 2.174229725863e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.174051695969e+01 true resid norm 2.174051695969e+01 ||r(i)||/||b|| 9.999181181770e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.172756288450e+01 true resid norm 2.172756288450e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.172607550829e+01 true resid norm 2.172607550829e+01 ||r(i)||/||b|| 9.999315442685e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.171411393798e+01 true resid norm 2.171411393798e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.171287087957e+01 true resid norm 2.171287087957e+01 ||r(i)||/||b|| 9.999427534358e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.170183476197e+01 true resid norm 2.170183476197e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.170079556874e+01 true resid norm 2.170079556874e+01 ||r(i)||/||b|| 9.999521149596e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.169062063291e+01 true resid norm 2.169062063291e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.168975161538e+01 true resid norm 2.168975161538e+01 ||r(i)||/||b|| 9.999599357922e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.168037662550e+01 true resid norm 2.168037662550e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.167964971448e+01 true resid norm 2.167964971448e+01 ||r(i)||/||b|| 9.999664714762e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.167101664634e+01 true resid norm 2.167101664634e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.167040844422e+01 true resid norm 2.167040844422e+01 ||r(i)||/||b|| 9.999719347673e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.166246255652e+01 true resid norm 2.166246255652e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.166195355024e+01 true resid norm 2.166195355024e+01 ||r(i)||/||b|| 9.999765028434e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.165464338097e+01 true resid norm 2.165464338097e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.165421750324e+01 true resid norm 2.165421750324e+01 ||r(i)||/||b|| 9.999803331913e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.164749647713e+01 true resid norm 2.164749647713e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.164714009809e+01 true resid norm 2.164714009809e+01 ||r(i)||/||b|| 9.999835371704e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.164096314641e+01 true resid norm 2.164096314641e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.164066488237e+01 true resid norm 2.164066488237e+01 ||r(i)||/||b|| 9.999862176173e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.163498993965e+01 true resid norm 2.163498993965e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.163474028123e+01 true resid norm 2.163474028123e+01 ||r(i)||/||b|| 9.999884604331e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.162952818847e+01 true resid norm 2.162952818847e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.162931918941e+01 true resid norm 2.162931918941e+01 ||r(i)||/||b|| 9.999903373271e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.162453352280e+01 true resid norm 2.162453352280e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.162440726736e+01 true resid norm 2.162440726736e+01 ||r(i)||/||b|| 9.999941614721e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.162067417859e+01 true resid norm 2.162067417859e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.162056845965e+01 true resid norm 2.162056845965e+01 ||r(i)||/||b|| 9.999951102847e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.161714328429e+01 true resid norm 2.161714328429e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.161705475368e+01 true resid norm 2.161705475368e+01 ||r(i)||/||b|| 9.999959046114e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.161391323696e+01 true resid norm 2.161391323696e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.161383909277e+01 true resid norm 2.161383909277e+01 ||r(i)||/||b|| 9.999965696081e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.161095817269e+01 true resid norm 2.161095817269e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.161089607130e+01 true resid norm 2.161089607130e+01 ||r(i)||/||b|| 9.999971263934e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.160825448313e+01 true resid norm 2.160825448313e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.160820246425e+01 true resid norm 2.160820246425e+01 ||r(i)||/||b|| 9.999975926384e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.160578062431e+01 true resid norm 2.160578062431e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.160573704698e+01 true resid norm 2.160573704698e+01 ||r(i)||/||b|| 9.999979830709e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.160351690469e+01 true resid norm 2.160351690469e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.160348039592e+01 true resid norm 2.160348039592e+01 ||r(i)||/||b|| 9.999983100543e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.160144534378e+01 true resid norm 2.160144534378e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.160141475452e+01 true resid norm 2.160141475452e+01 ||r(i)||/||b|| 9.999985839253e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159954951790e+01 true resid norm 2.159954951790e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159952388641e+01 true resid norm 2.159952388641e+01 ||r(i)||/||b|| 9.999988133324e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159781442369e+01 true resid norm 2.159781442369e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159779294497e+01 true resid norm 2.159779294497e+01 ||r(i)||/||b|| 9.999990055140e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159622635482e+01 true resid norm 2.159622635482e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159620835481e+01 true resid norm 2.159620835481e+01 ||r(i)||/||b|| 9.999991665207e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159477278300e+01 true resid norm 2.159477278300e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159475769728e+01 true resid norm 2.159475769728e+01 ||r(i)||/||b|| 9.999993014183e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159344225598e+01 true resid norm 2.159344225598e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159342961192e+01 true resid norm 2.159342961192e+01 ||r(i)||/||b|| 9.999994144493e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159222430526e+01 true resid norm 2.159222430526e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159221370694e+01 true resid norm 2.159221370694e+01 ||r(i)||/||b|| 9.999995091600e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159110935464e+01 true resid norm 2.159110935464e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159110047019e+01 true resid norm 2.159110047019e+01 ||r(i)||/||b|| 9.999995885138e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.159008863404e+01 true resid norm 2.159008863404e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.159008118520e+01 true resid norm 2.159008118520e+01 ||r(i)||/||b|| 9.999996549884e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158915410143e+01 true resid norm 2.158915410143e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158914785454e+01 true resid norm 2.158914785454e+01 ||r(i)||/||b|| 9.999997106468e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158829835518e+01 true resid norm 2.158829835518e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158829311446e+01 true resid norm 2.158829311446e+01 ||r(i)||/||b|| 9.999997572425e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158751461042e+01 true resid norm 2.158751461042e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158751021199e+01 true resid norm 2.158751021199e+01 ||r(i)||/||b|| 9.999997962514e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158679665551e+01 true resid norm 2.158679665551e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158679296227e+01 true resid norm 2.158679296227e+01 ||r(i)||/||b|| 9.999998289121e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158613880883e+01 true resid norm 2.158613880883e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158613570587e+01 true resid norm 2.158613570587e+01 ||r(i)||/||b|| 9.999998562521e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158553585461e+01 true resid norm 2.158553585461e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158553324556e+01 true resid norm 2.158553324556e+01 ||r(i)||/||b|| 9.999998791299e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158498299495e+01 true resid norm 2.158498299495e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158498079906e+01 true resid norm 2.158498079906e+01 ||r(i)||/||b|| 9.999998982679e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158447581795e+01 true resid norm 2.158447581795e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158447396778e+01 true resid norm 2.158447396778e+01 ||r(i)||/||b|| 9.999999142825e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158401029254e+01 true resid norm 2.158401029254e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158400873212e+01 true resid norm 2.158400873213e+01 ||r(i)||/||b|| 9.999999277051e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158358278593e+01 true resid norm 2.158358278593e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158358146892e+01 true resid norm 2.158358146892e+01 ||r(i)||/||b|| 9.999999389808e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158319004640e+01 true resid norm 2.158319004640e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158318893382e+01 true resid norm 2.158318893382e+01 ||r(i)||/||b|| 9.999999484516e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158282908258e+01 true resid norm 2.158282908258e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158282814200e+01 true resid norm 2.158282814200e+01 ||r(i)||/||b|| 9.999999564200e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158249719929e+01 true resid norm 2.158249719929e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158249640367e+01 true resid norm 2.158249640367e+01 ||r(i)||/||b|| 9.999999631362e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158219196835e+01 true resid norm 2.158219196835e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158219129508e+01 true resid norm 2.158219129508e+01 ||r(i)||/||b|| 9.999999688042e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158191118898e+01 true resid norm 2.158191118898e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158191061899e+01 true resid norm 2.158191061899e+01 ||r(i)||/||b|| 9.999999735895e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158165284721e+01 true resid norm 2.158165284721e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158165236453e+01 true resid norm 2.158165236453e+01 ||r(i)||/||b|| 9.999999776347e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158141511745e+01 true resid norm 2.158141511745e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158141470849e+01 true resid norm 2.158141470849e+01 ||r(i)||/||b|| 9.999999810502e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158119629651e+01 true resid norm 2.158119629651e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158119594981e+01 true resid norm 2.158119594981e+01 ||r(i)||/||b|| 9.999999839349e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158099482231e+01 true resid norm 2.158099482231e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158099452820e+01 true resid norm 2.158099452820e+01 ||r(i)||/||b|| 9.999999863719e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158080926117e+01 true resid norm 2.158080926117e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158080901147e+01 true resid norm 2.158080901147e+01 ||r(i)||/||b|| 9.999999884296e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158063828525e+01 true resid norm 2.158063828525e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158063807308e+01 true resid norm 2.158063807308e+01 ||r(i)||/||b|| 9.999999901686e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158048068325e+01 true resid norm 2.158048068325e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158048050284e+01 true resid norm 2.158048050284e+01 ||r(i)||/||b|| 9.999999916402e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158033535701e+01 true resid norm 2.158033535701e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158033520349e+01 true resid norm 2.158033520349e+01 ||r(i)||/||b|| 9.999999928862e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158020130005e+01 true resid norm 2.158020130005e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158020116932e+01 true resid norm 2.158020116932e+01 ||r(i)||/||b|| 9.999999939419e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.158007759139e+01 true resid norm 2.158007759139e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.158007747999e+01 true resid norm 2.158007747999e+01 ||r(i)||/||b|| 9.999999948381e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157996340108e+01 true resid norm 2.157996340108e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157996330609e+01 true resid norm 2.157996330609e+01 ||r(i)||/||b|| 9.999999955983e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157985795541e+01 true resid norm 2.157985795541e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157985787435e+01 true resid norm 2.157985787435e+01 ||r(i)||/||b|| 9.999999962437e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157976054754e+01 true resid norm 2.157976054754e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157976047831e+01 true resid norm 2.157976047831e+01 ||r(i)||/||b|| 9.999999967920e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157967053027e+01 true resid norm 2.157967053027e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157967047110e+01 true resid norm 2.157967047110e+01 ||r(i)||/||b|| 9.999999972582e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157958731219e+01 true resid norm 2.157958731219e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157958726157e+01 true resid norm 2.157958726157e+01 ||r(i)||/||b|| 9.999999976545e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157951034517e+01 true resid norm 2.157951034517e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157951030184e+01 true resid norm 2.157951030184e+01 ||r(i)||/||b|| 9.999999979917e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157943912577e+01 true resid norm 2.157943912577e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157943908862e+01 true resid norm 2.157943908862e+01 ||r(i)||/||b|| 9.999999982787e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157937319229e+01 true resid norm 2.157937319229e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157937316042e+01 true resid norm 2.157937316042e+01 ||r(i)||/||b|| 9.999999985232e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157931212167e+01 true resid norm 2.157931212167e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157931209430e+01 true resid norm 2.157931209430e+01 ||r(i)||/||b|| 9.999999987316e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157925552663e+01 true resid norm 2.157925552663e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157925550310e+01 true resid norm 2.157925550310e+01 ||r(i)||/||b|| 9.999999989096e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157920305267e+01 true resid norm 2.157920305267e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157920303242e+01 true resid norm 2.157920303242e+01 ||r(i)||/||b|| 9.999999990617e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157915437730e+01 true resid norm 2.157915437730e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157915435986e+01 true resid norm 2.157915435986e+01 ||r(i)||/||b|| 9.999999991916e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157910919689e+01 true resid norm 2.157910919689e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157910918184e+01 true resid norm 2.157910918184e+01 ||r(i)||/||b|| 9.999999993028e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157906723996e+01 true resid norm 2.157906723996e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157906722697e+01 true resid norm 2.157906722697e+01 ||r(i)||/||b|| 9.999999993978e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157902824830e+01 true resid norm 2.157902824830e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157902823705e+01 true resid norm 2.157902823705e+01 ||r(i)||/||b|| 9.999999994791e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157899198387e+01 true resid norm 2.157899198387e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157899197413e+01 true resid norm 2.157899197413e+01 ||r(i)||/||b|| 9.999999995487e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157895823195e+01 true resid norm 2.157895823195e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157895822349e+01 true resid norm 2.157895822349e+01 ||r(i)||/||b|| 9.999999996083e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157892678874e+01 true resid norm 2.157892678874e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157892678139e+01 true resid norm 2.157892678139e+01 ||r(i)||/||b|| 9.999999996594e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157889746679e+01 true resid norm 2.157889746679e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157889746039e+01 true resid norm 2.157889746039e+01 ||r(i)||/||b|| 9.999999997032e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157887009677e+01 true resid norm 2.157887009677e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157887009118e+01 true resid norm 2.157887009118e+01 ||r(i)||/||b|| 9.999999997409e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157884452362e+01 true resid norm 2.157884452362e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157884451873e+01 true resid norm 2.157884451873e+01 ||r(i)||/||b|| 9.999999997733e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157882060369e+01 true resid norm 2.157882060369e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157882059940e+01 true resid norm 2.157882059940e+01 ||r(i)||/||b|| 9.999999998012e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157879820540e+01 true resid norm 2.157879820540e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157879820163e+01 true resid norm 2.157879820163e+01 ||r(i)||/||b|| 9.999999998253e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157877720874e+01 true resid norm 2.157877720874e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157877720542e+01 true resid norm 2.157877720542e+01 ||r(i)||/||b|| 9.999999998461e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157875750190e+01 true resid norm 2.157875750190e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157875749897e+01 true resid norm 2.157875749897e+01 ||r(i)||/||b|| 9.999999998640e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157873897847e+01 true resid norm 2.157873897847e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157873897587e+01 true resid norm 2.157873897587e+01 ||r(i)||/||b|| 9.999999998795e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157872154251e+01 true resid norm 2.157872154251e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157872154020e+01 true resid norm 2.157872154020e+01 ||r(i)||/||b|| 9.999999998929e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157870510897e+01 true resid norm 2.157870510897e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157870510691e+01 true resid norm 2.157870510691e+01 ||r(i)||/||b|| 9.999999999046e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157868959888e+01 true resid norm 2.157868959888e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157868959704e+01 true resid norm 2.157868959704e+01 ||r(i)||/||b|| 9.999999999148e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157867494197e+01 true resid norm 2.157867494197e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157867494032e+01 true resid norm 2.157867494032e+01 ||r(i)||/||b|| 9.999999999237e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157866107352e+01 true resid norm 2.157866107352e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157866107205e+01 true resid norm 2.157866107205e+01 ||r(i)||/||b|| 9.999999999316e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157864793570e+01 true resid norm 2.157864793570e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157864793437e+01 true resid norm 2.157864793437e+01 ||r(i)||/||b|| 9.999999999384e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157863547316e+01 true resid norm 2.157863547316e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157863547196e+01 true resid norm 2.157863547196e+01 ||r(i)||/||b|| 9.999999999444e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157862363517e+01 true resid norm 2.157862363517e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157862363408e+01 true resid norm 2.157862363408e+01 ||r(i)||/||b|| 9.999999999497e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157861237790e+01 true resid norm 2.157861237790e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157861237692e+01 true resid norm 2.157861237692e+01 ||r(i)||/||b|| 9.999999999545e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157860166238e+01 true resid norm 2.157860166238e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157860166149e+01 true resid norm 2.157860166149e+01 ||r(i)||/||b|| 9.999999999586e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157859145093e+01 true resid norm 2.157859145093e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157859145011e+01 true resid norm 2.157859145011e+01 ||r(i)||/||b|| 9.999999999623e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157858170896e+01 true resid norm 2.157858170896e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157858170822e+01 true resid norm 2.157858170822e+01 ||r(i)||/||b|| 9.999999999656e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157857240396e+01 true resid norm 2.157857240396e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157857240328e+01 true resid norm 2.157857240328e+01 ||r(i)||/||b|| 9.999999999686e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157856350749e+01 true resid norm 2.157856350749e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157856350687e+01 true resid norm 2.157856350687e+01 ||r(i)||/||b|| 9.999999999713e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157855499672e+01 true resid norm 2.157855499672e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157855499615e+01 true resid norm 2.157855499615e+01 ||r(i)||/||b|| 9.999999999737e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157854685270e+01 true resid norm 2.157854685270e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157854685218e+01 true resid norm 2.157854685218e+01 ||r(i)||/||b|| 9.999999999759e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157853905689e+01 true resid norm 2.157853905689e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157853905641e+01 true resid norm 2.157853905641e+01 ||r(i)||/||b|| 9.999999999779e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157853159183e+01 true resid norm 2.157853159183e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157853159139e+01 true resid norm 2.157853159139e+01 ||r(i)||/||b|| 9.999999999797e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157852444358e+01 true resid norm 2.157852444358e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157852444318e+01 true resid norm 2.157852444318e+01 ||r(i)||/||b|| 9.999999999814e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157851759980e+01 true resid norm 2.157851759980e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157851759943e+01 true resid norm 2.157851759943e+01 ||r(i)||/||b|| 9.999999999830e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157851104819e+01 true resid norm 2.157851104819e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157851104785e+01 true resid norm 2.157851104785e+01 ||r(i)||/||b|| 9.999999999844e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157850477824e+01 true resid norm 2.157850477824e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157850477793e+01 true resid norm 2.157850477793e+01 ||r(i)||/||b|| 9.999999999857e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157849878133e+01 true resid norm 2.157849878133e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157849878105e+01 true resid norm 2.157849878105e+01 ||r(i)||/||b|| 9.999999999870e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157849305030e+01 true resid norm 2.157849305030e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157849305004e+01 true resid norm 2.157849305004e+01 ||r(i)||/||b|| 9.999999999881e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157848757711e+01 true resid norm 2.157848757711e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157848757687e+01 true resid norm 2.157848757687e+01 ||r(i)||/||b|| 9.999999999892e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157848235719e+01 true resid norm 2.157848235719e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157848235698e+01 true resid norm 2.157848235698e+01 ||r(i)||/||b|| 9.999999999902e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157847738622e+01 true resid norm 2.157847738622e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157847738603e+01 true resid norm 2.157847738603e+01 ||r(i)||/||b|| 9.999999999911e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157847265980e+01 true resid norm 2.157847265980e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157847265963e+01 true resid norm 2.157847265963e+01 ||r(i)||/||b|| 9.999999999920e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157846817397e+01 true resid norm 2.157846817397e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157846817382e+01 true resid norm 2.157846817382e+01 ||r(i)||/||b|| 9.999999999928e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157846392420e+01 true resid norm 2.157846392420e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157846392406e+01 true resid norm 2.157846392406e+01 ||r(i)||/||b|| 9.999999999936e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157845990653e+01 true resid norm 2.157845990653e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157845990640e+01 true resid norm 2.157845990640e+01 ||r(i)||/||b|| 9.999999999943e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157845611701e+01 true resid norm 2.157845611701e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157845611690e+01 true resid norm 2.157845611690e+01 ||r(i)||/||b|| 9.999999999949e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157845255034e+01 true resid norm 2.157845255034e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157845255025e+01 true resid norm 2.157845255025e+01 ||r(i)||/||b|| 9.999999999956e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157844920388e+01 true resid norm 2.157844920388e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157844920379e+01 true resid norm 2.157844920379e+01 ||r(i)||/||b|| 9.999999999961e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157844607422e+01 true resid norm 2.157844607422e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157844607415e+01 true resid norm 2.157844607415e+01 ||r(i)||/||b|| 9.999999999966e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157844315693e+01 true resid norm 2.157844315693e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157844315687e+01 true resid norm 2.157844315687e+01 ||r(i)||/||b|| 9.999999999971e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157844044827e+01 true resid norm 2.157844044827e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157844044822e+01 true resid norm 2.157844044822e+01 ||r(i)||/||b|| 9.999999999975e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157843794377e+01 true resid norm 2.157843794377e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157843794373e+01 true resid norm 2.157843794373e+01 ||r(i)||/||b|| 9.999999999979e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157843563819e+01 true resid norm 2.157843563819e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157843563815e+01 true resid norm 2.157843563815e+01 ||r(i)||/||b|| 9.999999999982e-01 -- Residual norms for sys_ solve. 0 KSP unpreconditioned resid norm 2.157843352838e+01 true resid norm 2.157843352838e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.157843352835e+01 true resid norm 2.157843352835e+01 ||r(i)||/||b|| 9.999999999985e-01 KSPGMRES, PC_LEFT =========================================================================== KSP Object:(sys_) 64 MPI processes type: gmres GMRES: restart=30, using Modified Gram-Schmidt Orthogonalization GMRES: happy breakdown tolerance 1e-30 maximum iterations=1, initial guess is zero tolerances: relative=0.1, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object:(sys_) 64 MPI processes type: shell Shell: no name linear system matrix = precond matrix: Matrix Object: 64 MPI processes type: shell rows=13599580, cols=13599580 Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.002292124139e+07 true resid norm 2.449788038787e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.326190403246e+06 true resid norm 2.531245015428e+01 ||r(i)||/||b|| 1.033250622238e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.739717855655e+07 true resid norm 2.371669076695e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.985543379911e+06 true resid norm 2.498578718744e+01 ||r(i)||/||b|| 1.053510687176e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.398796929309e+07 true resid norm 2.298427442616e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.704589856316e+06 true resid norm 2.460415108198e+01 ||r(i)||/||b|| 1.070477606810e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.089236032515e+07 true resid norm 2.235588179878e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.386114462569e+06 true resid norm 2.416221471778e+01 ||r(i)||/||b|| 1.080799001142e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.804685701080e+07 true resid norm 2.181662672264e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.064860185507e+06 true resid norm 2.378648886156e+01 ||r(i)||/||b|| 1.090291783600e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.537564640323e+07 true resid norm 2.135564314362e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.777612345074e+06 true resid norm 2.327085270069e+01 ||r(i)||/||b|| 1.089681661385e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.294279593659e+07 true resid norm 2.095726616172e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.488886320866e+06 true resid norm 2.279763848506e+01 ||r(i)||/||b|| 1.087815476940e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.098084576735e+07 true resid norm 2.060917986542e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.248376549327e+06 true resid norm 2.240008012911e+01 ||r(i)||/||b|| 1.086898182042e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.019561456782e+06 true resid norm 2.030700962013e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.085939671118e+06 true resid norm 2.185134911519e+01 ||r(i)||/||b|| 1.076049577163e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.300050101780e+06 true resid norm 2.002931432764e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.269346186616e+05 true resid norm 2.142563478176e+01 ||r(i)||/||b|| 1.069713841986e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.652884669299e+06 true resid norm 1.978066555523e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.640771390486e+05 true resid norm 2.081442904500e+01 ||r(i)||/||b|| 1.052261309756e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.353524975065e+06 true resid norm 1.953549132711e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.522143761649e+05 true resid norm 2.038404011524e+01 ||r(i)||/||b|| 1.043436265509e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.252964898432e+06 true resid norm 1.930842328583e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.888698887425e+05 true resid norm 1.996299465284e+01 ||r(i)||/||b|| 1.033900819208e+00 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.168021537227e+06 true resid norm 1.909903544303e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.273941824146e+05 true resid norm 1.909882021881e+01 ||r(i)||/||b|| 9.999887311469e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.586502807500e+06 true resid norm 1.887928409919e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.178978804995e+05 true resid norm 1.856683729467e+01 ||r(i)||/||b|| 9.834502832374e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.431338236002e+06 true resid norm 1.866755324771e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.555432571318e+05 true resid norm 1.787118772154e+01 ||r(i)||/||b|| 9.573395872721e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.582521011603e+06 true resid norm 1.843935677484e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.587356003840e+05 true resid norm 1.737293879753e+01 ||r(i)||/||b|| 9.421662051267e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.777657485855e+06 true resid norm 1.819422251827e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.564970760440e+05 true resid norm 1.692681114533e+01 ||r(i)||/||b|| 9.303398992913e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.999077015429e+06 true resid norm 1.794019518905e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.458049215411e+05 true resid norm 1.647145265048e+01 ||r(i)||/||b|| 9.181311840204e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.424910751426e+06 true resid norm 1.767662653672e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.905388652516e+05 true resid norm 1.589708321984e+01 ||r(i)||/||b|| 8.993278885439e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.715994507752e+06 true resid norm 1.739285389935e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.036420443800e+05 true resid norm 1.546755011591e+01 ||r(i)||/||b|| 8.893048952988e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.943690175475e+06 true resid norm 1.710685763894e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.110770046581e+05 true resid norm 1.504284205561e+01 ||r(i)||/||b|| 8.793457204766e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.106563874030e+06 true resid norm 1.681871238642e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.130852802303e+05 true resid norm 1.462182107711e+01 ||r(i)||/||b|| 8.693781510243e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.249647903316e+06 true resid norm 1.652805161407e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.237777317436e+05 true resid norm 1.421217230878e+01 ||r(i)||/||b|| 8.598818929560e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.505778194861e+06 true resid norm 1.623539189854e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.649340496776e+05 true resid norm 1.372091894621e+01 ||r(i)||/||b|| 8.451239755687e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.614023619525e+06 true resid norm 1.592895541303e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.580559265000e+05 true resid norm 1.331584489444e+01 ||r(i)||/||b|| 8.359521732068e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.612719298355e+06 true resid norm 1.561953501424e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.409861473171e+05 true resid norm 1.293121872461e+01 ||r(i)||/||b|| 8.278875595732e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.682898478437e+06 true resid norm 1.530924035448e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.378580277190e+05 true resid norm 1.254828687949e+01 ||r(i)||/||b|| 8.196544432606e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.684533662575e+06 true resid norm 1.499705950414e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.252042755207e+05 true resid norm 1.218146666248e+01 ||r(i)||/||b|| 8.122570067228e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.676724155194e+06 true resid norm 1.468434339290e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.125142514911e+05 true resid norm 1.181586893460e+01 ||r(i)||/||b|| 8.046576287719e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.620367728015e+06 true resid norm 1.437037340014e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.808303064703e+05 true resid norm 1.157893236061e+01 ||r(i)||/||b|| 8.057502778947e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.543322081458e+06 true resid norm 1.407046606575e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.593075952540e+05 true resid norm 1.124436574706e+01 ||r(i)||/||b|| 7.991466447893e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.558315700017e+06 true resid norm 1.377033063671e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.525329570906e+05 true resid norm 1.091483048471e+01 ||r(i)||/||b|| 7.926338715216e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.501953125956e+06 true resid norm 1.346970900090e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.250987417525e+05 true resid norm 1.061212582212e+01 ||r(i)||/||b|| 7.878511570969e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.443232987950e+06 true resid norm 1.317136990909e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.091549308229e+05 true resid norm 1.030998391524e+01 ||r(i)||/||b|| 7.827571457184e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.434870831821e+06 true resid norm 1.287467557686e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.996859001509e+05 true resid norm 1.002059602578e+01 ||r(i)||/||b|| 7.783183324472e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.270778963071e+06 true resid norm 1.258054858293e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.576251976483e+05 true resid norm 9.818167062907e+00 ||r(i)||/||b|| 7.804243986805e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.259919991031e+06 true resid norm 1.229892817656e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.477657910940e+05 true resid norm 9.534913132080e+00 ||r(i)||/||b|| 7.752637461735e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.172192156953e+06 true resid norm 1.201794272161e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.318821165868e+05 true resid norm 9.268263835589e+00 ||r(i)||/||b|| 7.712021974380e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.053034260242e+06 true resid norm 1.173921931783e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.090788264954e+05 true resid norm 9.020642294914e+00 ||r(i)||/||b|| 7.684192662803e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.034373397350e+06 true resid norm 1.146428878811e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.980500829475e+05 true resid norm 8.772973461987e+00 ||r(i)||/||b|| 7.652435858986e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.004418256680e+06 true resid norm 1.119280483666e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.841951522731e+05 true resid norm 8.519830722873e+00 ||r(i)||/||b|| 7.611881782278e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.901703208103e+06 true resid norm 1.092326556480e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.534918573625e+05 true resid norm 8.365143541591e+00 ||r(i)||/||b|| 7.658097747385e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.779906981032e+06 true resid norm 1.066752802537e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.384394939081e+05 true resid norm 8.144412112574e+00 ||r(i)||/||b|| 7.634769829713e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.752947655142e+06 true resid norm 1.041533213584e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.325198316312e+05 true resid norm 7.917864085967e+00 ||r(i)||/||b|| 7.602123468267e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.601890784580e+06 true resid norm 1.016542026399e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.143795455205e+05 true resid norm 7.722666750943e+00 ||r(i)||/||b|| 7.596997025590e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.576814867294e+06 true resid norm 9.920904934448e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.062745658802e+05 true resid norm 7.518974859238e+00 ||r(i)||/||b|| 7.578920379662e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.479362065138e+06 true resid norm 9.680474751189e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.830021040096e+05 true resid norm 7.403333537707e+00 ||r(i)||/||b|| 7.647696758672e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.453276712245e+06 true resid norm 9.454328940381e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.769713948431e+05 true resid norm 7.205675984388e+00 ||r(i)||/||b|| 7.621562598284e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.420658102754e+06 true resid norm 9.230656170662e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.689179448946e+05 true resid norm 7.012371831525e+00 ||r(i)||/||b|| 7.596829198137e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.381798306664e+06 true resid norm 9.009683706803e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.585112062893e+05 true resid norm 6.821739997113e+00 ||r(i)||/||b|| 7.571564351323e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.220455530106e+06 true resid norm 8.791330604159e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.440953633311e+05 true resid norm 6.654064814482e+00 ||r(i)||/||b|| 7.568893850192e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.190272125145e+06 true resid norm 8.577039200533e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.393910756559e+05 true resid norm 6.486692899451e+00 ||r(i)||/||b|| 7.562857937093e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.095597787750e+06 true resid norm 8.367095931284e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.191263466853e+05 true resid norm 6.405265620859e+00 ||r(i)||/||b|| 7.655303194158e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.070034339137e+06 true resid norm 8.171680944025e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.160349866917e+05 true resid norm 6.244100217255e+00 ||r(i)||/||b|| 7.641145389825e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.868248465769e+06 true resid norm 7.979246042756e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.986818842845e+05 true resid norm 6.111397264299e+00 ||r(i)||/||b|| 7.659116201646e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.856855773564e+06 true resid norm 7.792134975860e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.965876687782e+05 true resid norm 5.961833143292e+00 ||r(i)||/||b|| 7.651090698200e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.838299065304e+06 true resid norm 7.608397544412e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.934149405686e+05 true resid norm 5.816692415991e+00 ||r(i)||/||b|| 7.645095280625e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.813798278927e+06 true resid norm 7.428099200893e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.866811261665e+05 true resid norm 5.660243617741e+00 ||r(i)||/||b|| 7.620043115553e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.782955009713e+06 true resid norm 7.249607980355e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.832503284453e+05 true resid norm 5.519996301435e+00 ||r(i)||/||b|| 7.614199714513e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.697128864164e+06 true resid norm 7.074208920079e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.678408915526e+05 true resid norm 5.461841748483e+00 ||r(i)||/||b|| 7.720780952596e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.675173119033e+06 true resid norm 6.911947179044e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.658616776605e+05 true resid norm 5.329467567316e+00 ||r(i)||/||b|| 7.710515473085e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.648286216220e+06 true resid norm 6.752185425735e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.635009273971e+05 true resid norm 5.201437885076e+00 ||r(i)||/||b|| 7.703339818323e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.617441750669e+06 true resid norm 6.595076705042e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.583840611129e+05 true resid norm 5.068618832054e+00 ||r(i)||/||b|| 7.685458499942e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.583093377110e+06 true resid norm 6.439815953734e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.554852141056e+05 true resid norm 4.948536450529e+00 ||r(i)||/||b|| 7.684282417513e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.546190059323e+06 true resid norm 6.287556445923e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.518142544927e+05 true resid norm 4.831913836625e+00 ||r(i)||/||b|| 7.684883433148e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.507347991932e+06 true resid norm 6.138314859752e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.485349878036e+05 true resid norm 4.719705970390e+00 ||r(i)||/||b|| 7.688927789183e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.467165939355e+06 true resid norm 5.992268384007e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.451355088920e+05 true resid norm 4.611208646233e+00 ||r(i)||/||b|| 7.695263881270e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.426138416021e+06 true resid norm 5.849468139787e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.416581839486e+05 true resid norm 4.506268079239e+00 ||r(i)||/||b|| 7.703722751455e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.384716199542e+06 true resid norm 5.709993533895e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.381375465248e+05 true resid norm 4.404754648299e+00 ||r(i)||/||b|| 7.714114949784e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.343217801689e+06 true resid norm 5.573991762446e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.336766508402e+05 true resid norm 4.298834355166e+00 ||r(i)||/||b|| 7.712308410875e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.263318472124e+06 true resid norm 5.440643300800e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.229061513148e+05 true resid norm 4.258402282173e+00 ||r(i)||/||b|| 7.827019796624e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.232003833318e+06 true resid norm 5.317697008472e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.205732909782e+05 true resid norm 4.162024122374e+00 ||r(i)||/||b|| 7.826741756335e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.199738090351e+06 true resid norm 5.197158865745e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.181603724903e+05 true resid norm 4.068828287786e+00 ||r(i)||/||b|| 7.828947301581e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.167193419368e+06 true resid norm 5.079125553250e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.145097492727e+05 true resid norm 3.971508985598e+00 ||r(i)||/||b|| 7.819277046728e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.134138422293e+06 true resid norm 4.962980581716e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.120327011254e+05 true resid norm 3.883681905022e+00 ||r(i)||/||b|| 7.825301431422e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.101437419072e+06 true resid norm 4.849442186416e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.095501233318e+05 true resid norm 3.798716850114e+00 ||r(i)||/||b|| 7.833306809502e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.069277856788e+06 true resid norm 4.738577116081e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.070738547239e+05 true resid norm 3.716488594701e+00 ||r(i)||/||b|| 7.843047614629e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.037845966207e+06 true resid norm 4.630413655426e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.046150346447e+05 true resid norm 3.636866352352e+00 ||r(i)||/||b|| 7.854301198533e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.007234825617e+06 true resid norm 4.524959668423e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.021813999423e+05 true resid norm 3.559730545606e+00 ||r(i)||/||b|| 7.866877953514e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.775544860869e+05 true resid norm 4.422214818400e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.978202070196e+04 true resid norm 3.484975873129e+00 ||r(i)||/||b|| 7.880611902045e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.488609834218e+05 true resid norm 4.322172601371e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.695389359777e+04 true resid norm 3.413565002392e+00 ||r(i)||/||b|| 7.897798901666e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 9.214696118785e+05 true resid norm 4.224931074095e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.469902137132e+04 true resid norm 3.343044994122e+00 ||r(i)||/||b|| 7.912661616234e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.950870845934e+05 true resid norm 4.130303027271e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.249056720311e+04 true resid norm 3.274627825583e+00 ||r(i)||/||b|| 7.928299216694e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.697116035575e+05 true resid norm 4.038264984480e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.033257697530e+04 true resid norm 3.208233542281e+00 ||r(i)||/||b|| 7.944584009745e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.453211685827e+05 true resid norm 3.948772501683e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.832091875377e+04 true resid norm 3.135915021542e+00 ||r(i)||/||b|| 7.941493262034e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.216797599086e+05 true resid norm 3.860864783139e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.627413642680e+04 true resid norm 3.072738717485e+00 ||r(i)||/||b|| 7.958679959226e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.990572823054e+05 true resid norm 3.775421922124e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.429384370232e+04 true resid norm 3.011315430521e+00 ||r(i)||/||b|| 7.976103049236e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.775271519173e+05 true resid norm 3.692383052567e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.240098058835e+04 true resid norm 2.950876336403e+00 ||r(i)||/||b|| 7.991793631356e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.569577849408e+05 true resid norm 3.611590620066e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.021366473130e+04 true resid norm 2.883326560866e+00 ||r(i)||/||b|| 7.983536519466e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.372058650320e+05 true resid norm 3.531991730980e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.846201564563e+04 true resid norm 2.825437860461e+00 ||r(i)||/||b|| 7.999559669628e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.186823859844e+05 true resid norm 3.454565775181e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.645454100739e+04 true resid norm 2.768756790850e+00 ||r(i)||/||b|| 8.014775144078e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.009039532540e+05 true resid norm 3.379179961144e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.486783277284e+04 true resid norm 2.714470428131e+00 ||r(i)||/||b|| 8.032926506856e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.840035421923e+05 true resid norm 3.306002268451e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.338169307917e+04 true resid norm 2.661150947295e+00 ||r(i)||/||b|| 8.049452877543e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.678461027472e+05 true resid norm 3.234883797893e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.196913479431e+04 true resid norm 2.609287164254e+00 ||r(i)||/||b|| 8.066092407876e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 8.531343595954e+05 true resid norm 3.165764666136e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.956336475838e+04 true resid norm 2.521315542277e+00 ||r(i)||/||b|| 7.964317655217e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.292647185788e+05 true resid norm 3.093937251808e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.011128214676e+04 true resid norm 2.491404712539e+00 ||r(i)||/||b|| 8.052537946861e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.160959509194e+05 true resid norm 3.026711288751e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.875586200945e+04 true resid norm 2.444088996183e+00 ||r(i)||/||b|| 8.075064857579e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.815185589219e+05 true resid norm 2.961675578465e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.278556066066e+04 true resid norm 2.379210401763e+00 ||r(i)||/||b|| 8.033325523777e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.491615754964e+05 true resid norm 2.896460468227e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.941120293494e+04 true resid norm 2.333595475113e+00 ||r(i)||/||b|| 8.056714395767e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.579588092092e+05 true resid norm 2.833310835280e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.440154417880e+04 true resid norm 2.303284797087e+00 ||r(i)||/||b|| 8.129305010970e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 7.057244594752e+05 true resid norm 2.773775200151e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.536475866720e+04 true resid norm 2.246904523683e+00 ||r(i)||/||b|| 8.100528563239e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.287715404536e+05 true resid norm 2.714588693329e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.209681179962e+04 true resid norm 2.216634008273e+00 ||r(i)||/||b|| 8.165634866602e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.664516334566e+05 true resid norm 2.658629965864e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.097273715040e+04 true resid norm 2.170460851956e+00 ||r(i)||/||b|| 8.163832048175e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.420836267732e+05 true resid norm 2.603823752210e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.903864799610e+04 true resid norm 2.130605237826e+00 ||r(i)||/||b|| 8.182601591286e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 6.193302600381e+05 true resid norm 2.550665176560e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.704534961724e+04 true resid norm 2.092035652085e+00 ||r(i)||/||b|| 8.201921880262e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.976420790614e+05 true resid norm 2.499151338168e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.546930715790e+04 true resid norm 2.054795926720e+00 ||r(i)||/||b|| 8.221974777348e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.772898952473e+05 true resid norm 2.449292672055e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.440150776633e+04 true resid norm 2.021639958784e+00 ||r(i)||/||b|| 8.253974634594e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.547841050974e+05 true resid norm 2.401408442841e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.228612830103e+04 true resid norm 1.979081832781e+00 ||r(i)||/||b|| 8.241337864373e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.376605545187e+05 true resid norm 2.353824830851e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.055950684309e+04 true resid norm 1.944852018306e+00 ||r(i)||/||b|| 8.262518063434e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.214100388903e+05 true resid norm 2.307863909267e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.932210437804e+04 true resid norm 1.910951849107e+00 ||r(i)||/||b|| 8.280175626620e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.058714302270e+05 true resid norm 2.263359307558e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.816625925645e+04 true resid norm 1.877879065313e+00 ||r(i)||/||b|| 8.296866781343e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.909388121577e+05 true resid norm 2.220245909459e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.683093636919e+04 true resid norm 1.846200177286e+00 ||r(i)||/||b|| 8.315295929252e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.329998823527e+05 true resid norm 2.178553811154e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.764658710799e+04 true resid norm 1.808861909494e+00 ||r(i)||/||b|| 8.303039843368e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 5.127089481864e+05 true resid norm 2.137735868222e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.649958417254e+04 true resid norm 1.778557910112e+00 ||r(i)||/||b|| 8.319820687629e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.941990340779e+05 true resid norm 2.098271544633e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.540230497975e+04 true resid norm 1.748798414302e+00 ||r(i)||/||b|| 8.334471383243e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.765332435448e+05 true resid norm 2.060023230178e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.448913305182e+04 true resid norm 1.719948826180e+00 ||r(i)||/||b|| 8.349171994685e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.598768923070e+05 true resid norm 2.022998527969e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.367349522674e+04 true resid norm 1.691924815438e+00 ||r(i)||/||b|| 8.363450551478e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.441185294518e+05 true resid norm 1.987164947737e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.295086254071e+04 true resid norm 1.664720017972e+00 ||r(i)||/||b|| 8.377362029595e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.291593856471e+05 true resid norm 1.952496796024e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.223021837007e+04 true resid norm 1.638485564698e+00 ||r(i)||/||b|| 8.391745215840e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 4.085759008593e+05 true resid norm 1.918983748348e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.970724068852e+04 true resid norm 1.610952094581e+00 ||r(i)||/||b|| 8.394818851214e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.957620749449e+05 true resid norm 1.886222568820e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.898689575863e+04 true resid norm 1.586020603508e+00 ||r(i)||/||b|| 8.408448874091e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.834217221234e+05 true resid norm 1.854589698767e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.820595775150e+04 true resid norm 1.562013755714e+00 ||r(i)||/||b|| 8.422422257343e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.715334666535e+05 true resid norm 1.824073602450e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.749144042261e+04 true resid norm 1.538734973139e+00 ||r(i)||/||b|| 8.435706602370e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.600573676075e+05 true resid norm 1.794640301052e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.694536665814e+04 true resid norm 1.516013466664e+00 ||r(i)||/||b|| 8.447450253825e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.489263161490e+05 true resid norm 1.766239187375e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.644564069678e+04 true resid norm 1.494006596876e+00 ||r(i)||/||b|| 8.458687858107e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.380248715629e+05 true resid norm 1.738841235180e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.597555438996e+04 true resid norm 1.472697677485e+00 ||r(i)||/||b|| 8.469420023461e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.667659247721e+05 true resid norm 1.712424571470e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.654252119480e+04 true resid norm 1.456001910831e+00 ||r(i)||/||b|| 8.502575442383e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.522015594938e+05 true resid norm 1.687212156903e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.596138857678e+04 true resid norm 1.435996865255e+00 ||r(i)||/||b|| 8.511062816727e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.382009852103e+05 true resid norm 1.662887023898e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.541846169028e+04 true resid norm 1.416555058016e+00 ||r(i)||/||b|| 8.518648817735e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.798035449734e+05 true resid norm 1.639439971116e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.775345622551e+04 true resid norm 1.395931502986e+00 ||r(i)||/||b|| 8.514685060634e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.613921495049e+05 true resid norm 1.616423964217e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.743150028153e+04 true resid norm 1.378512077811e+00 ||r(i)||/||b|| 8.528159123643e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.356994988466e+05 true resid norm 1.594340402057e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.506596460379e+04 true resid norm 1.360303797588e+00 ||r(i)||/||b|| 8.532078819759e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.206346919057e+05 true resid norm 1.572943344035e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.434672197720e+04 true resid norm 1.343368476274e+00 ||r(i)||/||b|| 8.540475926032e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 3.063520836504e+05 true resid norm 1.552353359558e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.367414545423e+04 true resid norm 1.327016733209e+00 ||r(i)||/||b|| 8.548419243843e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.925462634954e+05 true resid norm 1.532546815866e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.296484647842e+04 true resid norm 1.311124848703e+00 ||r(i)||/||b|| 8.555202589106e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.796269808233e+05 true resid norm 1.513506025755e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.236050537716e+04 true resid norm 1.295839678441e+00 ||r(i)||/||b|| 8.561840233140e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.673886535798e+05 true resid norm 1.495205690876e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.182543921288e+04 true resid norm 1.281043033911e+00 ||r(i)||/||b|| 8.567670934696e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.557837875988e+05 true resid norm 1.477613168990e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.131257469770e+04 true resid norm 1.266876951660e+00 ||r(i)||/||b|| 8.573806583802e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.675278069922e+05 true resid norm 1.460719526562e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.160603922336e+04 true resid norm 1.253134495738e+00 ||r(i)||/||b|| 8.578885083351e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.540205307633e+05 true resid norm 1.444475156985e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.102403005834e+04 true resid norm 1.239431346704e+00 ||r(i)||/||b|| 8.580496111067e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.413933374247e+05 true resid norm 1.428827657938e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.048999459623e+04 true resid norm 1.226568946489e+00 ||r(i)||/||b|| 8.584442914969e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.295119387968e+05 true resid norm 1.413798173390e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.998525341569e+04 true resid norm 1.214114300457e+00 ||r(i)||/||b|| 8.587606939296e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.426436147259e+05 true resid norm 1.399360438391e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.061485597116e+04 true resid norm 1.199471088761e+00 ||r(i)||/||b|| 8.571566380284e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.291137137931e+05 true resid norm 1.385157105880e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.997559455883e+04 true resid norm 1.187549761533e+00 ||r(i)||/||b|| 8.573393996187e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.165612328469e+05 true resid norm 1.371506164050e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.939565247596e+04 true resid norm 1.176493841104e+00 ||r(i)||/||b|| 8.578115592494e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 2.048408604369e+05 true resid norm 1.358430111565e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.884655241270e+04 true resid norm 1.165686483196e+00 ||r(i)||/||b|| 8.581129594173e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.938887692107e+05 true resid norm 1.345886911627e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.835195733928e+04 true resid norm 1.155381453428e+00 ||r(i)||/||b|| 8.584535917893e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.973039893100e+05 true resid norm 1.333859485126e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.855817703015e+04 true resid norm 1.149964427844e+00 ||r(i)||/||b|| 8.621331112214e-01 -- Residual norms for sys_ solve. 0 KSP preconditioned resid norm 1.862683499978e+05 true resid norm 1.322563930533e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.797803334707e+04 true resid norm 1.140369682865e+00 ||r(i)||/||b|| 8.622416327398e-01 dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From fd.kong at siat.ac.cn Fri Jun 1 04:24:22 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Fri, 1 Jun 2012 17:24:22 +0800 Subject: [petsc-users] How to free SieveMesh memory? Message-ID: Hi guys, In directory: /src//src/dm/impls/mesh/, we can find a file named "mesh.c", where a function called "DMDestroy_Mesh(DM dm)", which has the following code: PetscErrorCode DMDestroy_Mesh(DM dm) { DM_Mesh *mesh = (DM_Mesh *) dm->data; mesh->m = PETSC_NULL; VecScatterDestroy(&mesh->globalScatter); return(0); } When we destroy SieveMesh, this function will be called. In this function, we just set "mesh->m = PETSC_NULL", Whether memory occupied by SieveMesh will be free? I don't think so! Thus, there are anyone who have an idea on how to free SieveMesh memory. Regards, ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 1 06:53:00 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 1 Jun 2012 06:53:00 -0500 Subject: [petsc-users] GMRES pc right fails while pc left succeeds In-Reply-To: References: Message-ID: On Fri, Jun 1, 2012 at 3:36 AM, Klaij, Christiaan wrote: > To solve the incompressible Navier-Stokes equations, I'm using > cell-centered finite volumes and Picard linearization. At each > non-linear iteration, I'm using one linear (Krylov) iteration with a > SIMPLE-type block preconditioner (for example Elman e.a, JCP 227, > 2008). The matrix and preconditioner are shells. Below, I'm showing > the results for the first few non-linear iterations using > > (1) KSPRICHARDSON to check that the shells are ok > (2) GMRES with right preconditioning > (3) GMRES with left preconditioning > > To my surprise (2) fails while (1) and (3) succeed. The reason I'm > interested in right preconditioning is that SIMPLE is a variable > preconditioner so in theory I should be using FGMRES as soon as I > increase the number of linear (Krylov) iterations. > One inner iteration is already nonlinear. What is the inner iteration here? SIMPLE uses a diagonal approximation of the (velocity,velocity) block in an explicit approximation of the Schur complement (which you could form using the matrix product). Is there a reason you aren't using PCFIELDSPLIT for this? > What could be the > reason that GMRES fails with right pc? > Both might be failing, you'll have to show that a tight tolerance can eventually be reached. What does FGMRES do? Note that the preconditioned norm (see Richardson and left-GMRES) becomes huge. A preconditioned residual that is 10^6 larger than unpreconditioned usually means the preconditioner is incorrect. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbakosi at lanl.gov Fri Jun 1 09:27:25 2012 From: jbakosi at lanl.gov (Jozsef Bakosi) Date: Fri, 1 Jun 2012 08:27:25 -0600 Subject: [petsc-users] Is PCFactorSetZeroPivot() effective? In-Reply-To: References: <20120530232001.GF19605@karman> <20120531200821.GV19605@karman> <20120531204751.GC19605@karman> <20120531222643.GI19605@karman> Message-ID: <20120601142724.GD25937@karman> On 05.31.2012 18:05, Jed Brown wrote: > On Thu, May 31, 2012 at 5:26 PM, Jozsef Bakosi wrote: > > > Great. Thanks. When I pass these options on the command line, how can I > > get a feedback to see if the option has actually been used or not? > > > > -options_left shows whether the option was queried > > -ksp_view will have a line "tolerance for zero pivot" telling you what was > actually used > > > > > > > What discretization are you using? > > > > > > > > Straightforward Galerkin FEM Poisson solver. > > > > > > > > > > What boundary conditions? I'm guessing the constant is in the null space, > > > in which case the coarse space really should be singular and you should > > not > > > be trying to solve it with a direct solver. > > > > I do set the additive constant. I have no problem with this on lower > > core-counts. > > > > What do you mean by "set the additive constant"? How do you enforce it? > Best option is usually to tell the KSP about the null space and use an > inexact coarse level solver. If you use ML with default options, the coarse > levels are generally small enough that you can brute-force it by using > -mg_coarse_redundant_pc_type svd. If you need big coarse spaces, you'll > probably need to use an inexact coarse level solver (possibly use a shift, > as in -pc_coarse_redundant_pc_factor_shift_type nonzero. I mean I grab one node with a Dirichlet condition and apply a Neumann on all boundaries, i.e. the problem is well-posed: the additive constant, up to which the solution is determined, is accounted for. > > I did another test and set the PETSC_OPTIONS environment variable to > > '-options_table -options_left', then I get: > > > > ... > > > > [49]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12! > > > > This is an error. How do you get to the output shown below? I'm not sure if I understand your question. I set PETSC_OPTIONS environment variable to '-options_table -options_left'. > > > ... > > > > #PETSc Option Table entries: > > -mg_levels_ksp_norm_type no > > -mg_levels_ksp_richardson_scale 0.90 > > -mg_levels_ksp_type richardson > > -mg_levels_pc_type bjacobi > > -mg_levels_sub_ksp_norm_type no > > -mg_levels_sub_ksp_type preonly > > -mg_levels_sub_pc_type icc > > -options_left > > -options_table > > -pc_asm_overlap 1 > > -pc_mg_cycle_type v > > -pc_mg_smoothdown 1 > > -pc_mg_smoothup 1 > > -pc_ml_maxCoarseSize 1500 > > -pc_ml_maxNlevels 10 > > -pc_type asm > > -sub_pc_factor_levels 0 > > -sub_pc_factor_zeropivot 1e-20 > > -sub_pc_type ilu > > #End of PETSc Option Table entries > > There are no unused options. > > > > So how do I tell, which zeropivot setting (the 1e-20 or the 1e-12) is the > > one petsc is using? It seems like your suggestion -mg_coarse_redundant_pc_factor_zeropivot <2.22045e-14>: Pivot is considered zero if less than (PCFactorSetZeroPivot) remedies the problem as I'm not getting the zeropivot error anymore. I guess the problem was at the coarsest MG level. I set mg_coarse_redundent_pc_factor to 1.0e-20 and that helped. Thanks, Jozsef From jedbrown at mcs.anl.gov Fri Jun 1 09:35:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 1 Jun 2012 09:35:48 -0500 Subject: [petsc-users] Is PCFactorSetZeroPivot() effective? In-Reply-To: <20120601142724.GD25937@karman> References: <20120530232001.GF19605@karman> <20120531200821.GV19605@karman> <20120531204751.GC19605@karman> <20120531222643.GI19605@karman> <20120601142724.GD25937@karman> Message-ID: On Fri, Jun 1, 2012 at 9:27 AM, Jozsef Bakosi wrote: > I mean I grab one node with a Dirichlet condition and apply a Neumann on > all boundaries, i.e. the problem is well-posed: the additive constant, > up to which the solution is determined, is accounted for. > This is fragile because smoothed aggregation can "coarsen out" this constraint. We usually recommend informing the KSP of the null space. See the section of the user's manual on solving singular systems. > > > > I did another test and set the PETSC_OPTIONS environment variable to > > > '-options_table -options_left', then I get: > > > > > > ... > > > > > > [49]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12! > > > > > > > This is an error. How do you get to the output shown below? > > I'm not sure if I understand your question. I set PETSC_OPTIONS > environment variable to '-options_table -options_left'. > You must not have been checking errors or something because the code evidently continued past the error, getting all the way to PetscFinalize(). > It seems like your suggestion > > -mg_coarse_redundant_pc_factor_zeropivot <2.22045e-14>: Pivot is > considered zero if less than (PCFactorSetZeroPivot) > > remedies the problem as I'm not getting the zeropivot error anymore. I > guess the problem was at the coarsest MG level. I set > mg_coarse_redundent_pc_factor to 1.0e-20 and that helped. > Yes, but is the convergence good now? -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at risoe.dtu.dk Fri Jun 1 09:47:24 2012 From: frtr at risoe.dtu.dk (Frederik Treue) Date: Fri, 01 Jun 2012 16:47:24 +0200 Subject: [petsc-users] How to use external vectors in a KSPmonitor Message-ID: <4FC8D5FC.2090203@risoe.dtu.dk> Hi, I need to access a vector from outside the ksp context in my own, user-defined KSPmonitor. I have tried using the "context" input variable in the KSPMonitor function to pass a pointer to the relevant structure, but I get segmentation faults: Can this "context" variable be used for this, and if no, how does one do? /Frederik Treue From jedbrown at mcs.anl.gov Fri Jun 1 09:52:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 1 Jun 2012 09:52:38 -0500 Subject: [petsc-users] How to use external vectors in a KSPmonitor In-Reply-To: <4FC8D5FC.2090203@risoe.dtu.dk> References: <4FC8D5FC.2090203@risoe.dtu.dk> Message-ID: On Fri, Jun 1, 2012 at 9:47 AM, Frederik Treue wrote: > I need to access a vector from outside the ksp context in my own, > user-defined KSPmonitor. I have tried using the "context" input variable in > the KSPMonitor function to pass a pointer to the relevant structure, but I > get segmentation faults: Can this "context" variable be used for this, and > if no, how does one do? This is what the context is for. Send the complete output from the SEGV and any relevant snippets of code showing how you use it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 1 10:26:48 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 1 Jun 2012 11:26:48 -0400 Subject: [petsc-users] How to free SieveMesh memory? In-Reply-To: References: Message-ID: On Fri, Jun 1, 2012 at 5:24 AM, fdkong wrote: > Hi guys, > > In directory: /src//src/dm/impls/mesh/, we can find a file named "mesh.c", > where a function called "DMDestroy_Mesh(DM dm)", which has the following > code: > > PetscErrorCode DMDestroy_Mesh(DM dm) > { > DM_Mesh *mesh = (DM_Mesh *) dm->data; > > mesh->m = PETSC_NULL; > VecScatterDestroy(&mesh->globalScatter); > return(0); > } > > When we destroy SieveMesh, this function will be called. In this > function, we just set "mesh->m = PETSC_NULL", Whether memory occupied by > SieveMesh will be free? I don't think so! > 1) It will be freed because it is allocated in C++, and the destructor is automatically called by the smart pointer when it is set to NULL 2) This is my old code. It works, but you should seriously consider using the new revision which is all in C and easier to understand, and better integrated with the solvers. See http://www.mcs.anl.gov/petsc/petsc-dev/src/snes/examples/tutorials/ex62.c.html for the Stokes problem on an unstructured grid. Matt > Thus, there are anyone who have an idea on how to free SieveMesh memory. > > Regards, > ** > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > ** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbakosi at lanl.gov Fri Jun 1 14:27:34 2012 From: jbakosi at lanl.gov (Jozsef Bakosi) Date: Fri, 1 Jun 2012 13:27:34 -0600 Subject: [petsc-users] Is PCFactorSetZeroPivot() effective? In-Reply-To: References: <20120530232001.GF19605@karman> <20120531200821.GV19605@karman> <20120531204751.GC19605@karman> <20120531222643.GI19605@karman> <20120601142724.GD25937@karman> Message-ID: <20120601192734.GI25937@karman> On 06.01.2012 09:35, Jed Brown wrote: > On Fri, Jun 1, 2012 at 9:27 AM, Jozsef Bakosi wrote: > > > It seems like your suggestion > > > > -mg_coarse_redundant_pc_factor_zeropivot <2.22045e-14>: Pivot is > > considered zero if less than (PCFactorSetZeroPivot) > > > > remedies the problem as I'm not getting the zeropivot error anymore. I > > guess the problem was at the coarsest MG level. I set > > mg_coarse_redundent_pc_factor to 1.0e-20 and that helped. > > > > Yes, but is the convergence good now? Yeah. It seems like I get a similar convergence that I was getting on lower core counts: 20-30 MG iterations on a 80 million cell mesh. Jozsef From zonexo at gmail.com Sat Jun 2 04:27:48 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 02 Jun 2012 11:27:48 +0200 Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in ksp In-Reply-To: References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com> <4FA3F056.4090406@gmail.com> <4FA4279F.1030204@gmail.com> <5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov> <4FA42CFC.8090002@gmail.com> <4FA44D5C.6010205@gmail.com> <4FA97132.3080103@gmail.com> <4FAD12FB.6080506@gmail.com> <4FAD195B.4030302@gmail.com> Message-ID: <4FC9DC94.7050909@gmail.com> Hi, I have modified the C version of the ex29 example in ksp to Fortran. I have attached it. Feel free to include it in the distribution of PETSc. The DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90 are not working in linux with ifort, so VecSetValues was used. Worked in VS2008 though. Btw, I need to solve the Poisson equation multiple times using different RHS vector. Which subroutines do I need to call only once? Can I also add : call MatSetOption(jac,MAT_STRUCTURALLY_SYMMETRIC,PETSC_TRUE,ierr) call MatSetOption(jac,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE,ierr) call MatSetOption(jac,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE,ierr) call KSPSetOperators(ksp,A_mat,A_mat,SAME_NONZERO_PATTERN,ierr) Yours sincerely, TAY wee-beng On 11/5/2012 4:04 PM, John Mousel wrote: > Include both header files. > > #include > #include > > The h90 header only includes the interface definitions for Fortran. > > John > > On Fri, May 11, 2012 at 8:51 AM, TAY wee-beng > wrote: > > > > On 11/5/2012 3:30 PM, Matthew Knepley wrote: >> On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng > > wrote: >> >> Hi, >> >> I have been using the GUI environment to do debugging so I am >> a bit reluctant to learn Valgrind as its outputs seems a bit >> daunting.? But I guess John is right. I've been spending >> these few days learning bit by bit. >> >> I realised that the error occurs in computerhs, at: >> >> >> I bet this is a beautiful Fortranism. Do you include the F90 >> header file with the interface definition? >> If not, Fortran just craps out like this. I can't stress enough >> how much time would be saved by >> switching languages to something with at least a modicum of error >> checking. > > I initially used: > > #include "finclude/petsc.h90" > > Compilation and linking was fine in Linux and vs2008. > > Now I changed it to what ex22.F was using : > > #include > #include > #include > #include > #include > #include > > Compiling was ok but linking failed in Linux and VS2008: > > undefined reference to `dmdavecgetarrayf90_' > > I tried changing #include to #include > and everything was ok in VS2008 again, > giving the right answers. > > However, in Linux, I got the following error: > > [wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c? -fPIC > -g? ? -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include > -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include > -I/opt/openmpi-1.5.3/include? ? ? -o ex29f.o ex29f.F90 > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDAStencilType? :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDAStencilType? :: st > --------^ > ex29f.F90(68): error #6404: This name does not have a type, and > must have an explicit type.? ? [DMDA_BOUNDARY_NONE] > call > DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > -----------------------------------^ > ex29f.F90(68): error #6404: This name does not have a type, and > must have an explicit type.? ? [DMDA_STENCIL_STAR] > call > DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > -------------------------------------------------------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDAStencilType? :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDAStencilType? :: st > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > ? ? ? ? ? ? ? DMDAStencilType? :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > ? ? ? ? ? ? ? DMDAStencilType? :: st > > Is there some errors in petscdmda.h90? > > > >> >> ? ? Matt >> ? >> >> *call DMDAVecRestoreArrayF90(da,b,array,ierr)* >> >> ==27464== Invalid write of size 8 >> ==27464==? ? ? at 0x402835: computerhs_ (ex29f.F90:119) >> ==27464==? Address 0xfffffffffffffef0 is not stack'd, >> malloc'd or (recently) free'd >> ==27464== >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >> Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: ---------------------? Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are >> not available, >> [0]PETSC ERROR:? ? ? ? ? ? INSTEAD the line number of the >> start of the function >> [0]PETSC ERROR:? ? ? ? ? ? is given. >> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown >> [0]PETSC ERROR: [0] DMComputeFunction line 2085 >> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c >> [0]PETSC ERROR: [0] KSPSetUp line 182 >> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> >> I have checked that "array" 's values are correct. This >> statement executed without problems in VS2008. If I replace >> the above with something like: >> >> *call VecSet(b,Hy,ierr)* >> >> Everything is fine in Linux. >> >> Is there something wrong with *DMDAVecRestoreArrayF90*? >> >> My code in the area is: >> >> call >> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) >> >> call DMDAVecGetArrayF90(da,b,array,ierr) >> >> do j = ys,ys+ym-1 >> >> ? ? ? do i = xs,xs+xm-1 >> >> ? ? ? ? ? ? array(i,j) = >> exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy >> >> ? ? ? end do >> >> end do >> >> call DMDAVecRestoreArrayF90(da,b,array,ierr) >> >> call VecAssemblyBegin(b,ierr) >> >> call VecAssemblyEnd(b,ierr) >> >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> On 8/5/2012 9:41 PM, John Mousel wrote: >>> TAY wee-bing, >>> >>> If you want to be a programmer that writes interesting and >>> reliable code, you need to be willing to use the tools of >>> the trade. I can't think of a bigger time-saver than >>> Valgrind. I would suggest that you learn to use it and use >>> it a lot. I bet it will lead you to the root of your problem >>> pretty quickly. >>> >>> John >>> >>> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng >>> > wrote: >>> >>> Hi, >>> >>> I compiled and run my code under visual studio 2008 with >>> intel fortran. Everything works ok. >>> >>> However, when I tried to run the code in linux, I got >>> the error as below. The error happens when >>> KSPSetUp(ksp,ierr) is called. >>> >>> However, I am not able to print VecView or MatView to >>> view if there's any errors. Is there any recommendation >>> for debugging? I hope I do not need to valgrind if possible. >>> >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: >>> Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC >>> ERROR: or try http://valgrind.org on GNU/linux and Apple >>> Mac OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in >>> stack below >>> [0]PETSC ERROR: ---------------------? Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the >>> stack are not available, >>> [0]PETSC ERROR:? ? ? ? ? ? INSTEAD the line number of >>> the start of the function >>> [0]PETSC ERROR:? ? ? ? ? ? is given. >>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown >>> [0]PETSC ERROR: [0] DMComputeFunction line 2085 >>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c >>> [0]PETSC ERROR: [0] KSPSetUp line 182 >>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: >>> 7ecdd63ec420b1659b960e65d96e822c5ac1a968? HG Date: Mon >>> May 07 21:42:26 2012 -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent >>> updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about >>> trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by >>> wtay Tue May? 8 20:45:42 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib >>> [0]PETSC ERROR: Configure run at Tue May? 8 10:47:59 2012 >>> [0]PETSC ERROR: Configure options >>> --with-mpi-dir=/opt/openmpi-1.5.3/ >>> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ >>> --with-debugging=1 --download-hypre=1 >>> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug >>> --known-mpi-shared=1 --with-shared-libraries >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in >>> unknown directory unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator >>> MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI >>> processes. >>> You may or may not see output from other processes, >>> depending on >>> exactly when Open MPI kills them. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> On 5/5/2012 1:43 AM, Matthew Knepley wrote: >>>> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng >>>> > wrote: >>>> >>>> Hi, >>>> >>>> I wonder if you people are interested to include my >>>> ex29 fortran version in the petsc examples, which >>>> can help people who are using fortran. >>>> >>>> >>>> Yes, we will definitely include it. Please send the >>>> source, and a representative output with run options. >>>> >>>> ? Thanks, >>>> >>>> ? ? ? Matt >>>> ? >>>> >>>> Thanks. >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> On 4/5/2012 9:28 PM, Matthew Knepley wrote: >>>>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng >>>>> > wrote: >>>>> >>>>> >>>>> On 4/5/2012 9:16 PM, Barry Smith wrote: >>>>> >>>>> ? ? Do an hg pull and then run make in >>>>> src/mat/interface/ftn-custom/ >>>>> >>>>> ? ? Then it should link. >>>>> >>>>> ? ? Barry >>>>> >>>>> ? ? There was a E missing from the all >>>>> caps name of the function. >>>>> >>>>> After hg pull, I did: >>>>> >>>>> cd src//mat/interface/ftn-custom/ >>>>> >>>>> User at User-PC >>>>> /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom >>>>> $ make >>>>> make[1]: Warning: File >>>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)' >>>>> has modification time 787 s in the future >>>>> make[1]: Nothing to be done for `libc'. >>>>> make[1]: warning: ? Clock skew detected. >>>>> ? Your build may be incomplete. >>>>> >>>>> But it still can't work. >>>>> >>>>> >>>>> Something is messed up with the clock on this machine. >>>>> >>>>> HOWEVER, development requires certain basic skills >>>>> in order to debug your work. We >>>>> cannot be the ones debugging your code. Now >>>>> >>>>> ? nm $PETSC_ARCH/lib/libpetsc.a | grep -i >>>>> MatNullSpaceRemove >>>>> >>>>> ? will look for the symbol. >>>>> >>>>> ? ? Matt >>>>> ? >>>>> >>>>> >>>>> >>>>> On May 4, 2012, at 2:11 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Fri, May 4, 2012 at 3:01 PM, TAY >>>>> wee-beng>>>> > ? wrote: >>>>> >>>>> On 4/5/2012 5:17 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>> On Fri, May 4, 2012 at 11:05 AM, >>>>> TAY wee-beng>>>> > ? wrote: >>>>> >>>>> On 4/5/2012 3:05 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Fri, May 4, 2012 at 8:59 >>>>> AM, TAY >>>>> wee-beng>>>> > >>>>> ? wrote: >>>>> >>>>> Hi, >>>>> >>>>> Is there anything else I can >>>>> try to get it working right? >>>>> >>>>> The MatGetNullSpaceRemove() is >>>>> missing. >>>>> >>>>> Where should I add >>>>> MatGetNullSpaceRemove and what are >>>>> its syntax? I googled but there's >>>>> no results. >>>>> >>>>> Fixed in p;etsc-dev: >>>>> >>>>> ? >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html >>>>> >>>>> I just compiled the updated petsc-dev >>>>> but I got the same error msg when I use: >>>>> >>>>> call >>>>> MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr) >>>>> >>>>> error LNK2019: unresolved external >>>>> symbol MATNULLSPACEREMOVE referenced >>>>> in function COMPUTERHS >>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : >>>>> fatal error LNK1120: 1 unresolved >>>>> externals >>>>> >>>>> That function is in: >>>>> >>>>> ? src/mat/interface/ftn-custom/zmatrixf.c >>>>> >>>>> Did that compile? Can you see the >>>>> symbol in libpetsc.a? >>>>> >>>>> ? ? Matt >>>>> >>>>> ? ? ? Matt >>>>> >>>>> Thanks. >>>>> >>>>> ? ? Matt >>>>> >>>>> Thanks??? >>>>> >>>>> >>>>> On 2/5/2012 10:11 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Wed, May 2, 2012 at >>>>> 1:55 PM, TAY >>>>> wee-beng>>>> > >>>>> ? wrote: >>>>> Hi, >>>>> >>>>> I did a MatView and >>>>> VecView on both C and >>>>> Fortran, right after Mat >>>>> and Vec assembly. I have >>>>> attached the printout >>>>> below. They are exactly >>>>> the same, but yet the >>>>> result is different in >>>>> Neumann condition. >>>>> However, the dirichlet >>>>> condition gives the >>>>> correct ans. Is there >>>>> anything else that could >>>>> be wrong even if the Mat >>>>> and Vec are the same? >>>>> >>>>> Did you set the null space >>>>> for the matrix when you >>>>> have Neumann conditions? >>>>> >>>>> Yes, for the matrix, I set as: >>>>> >>>>> call >>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) >>>>> >>>>> ? ? call >>>>> MatSetNullSpace(jac,nullspace,ierr) >>>>> >>>>> ? ? call >>>>> MatNullSpaceDestroy(nullspace,ierr) >>>>> >>>>> for the Vec, >>>>> >>>>> call >>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) >>>>> >>>>> ? ? !call >>>>> MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr) >>>>> >>>>> ? ? call >>>>> MatNullSpaceDestroy(nullspace,ierr) >>>>> >>>>> MatNullSpaceRemove was comment >>>>> out because there's error >>>>> during linking >>>>> >>>>> >>>>> ? ? Matt >>>>> >>>>> Thanks! >>>>> >>>>> Fortran: >>>>> >>>>> Matrix Object: 1 MPI processes >>>>> ? type: seqaij >>>>> row 0: (0, 2) ? (1, -1) >>>>> ? (3, -1) >>>>> row 1: (0, -1) ? (1, 3) >>>>> ? (2, -1) ? (4, -1) >>>>> row 2: (1, -1) ? (2, 2) >>>>> ? (5, -1) >>>>> row 3: (0, -1) ? (3, 3) >>>>> ? (4, -1) ? (6, -1) >>>>> row 4: (1, -1) ? (3, -1) >>>>> ? (4, 4) ? (5, -1) ? (7, -1) >>>>> row 5: (2, -1) ? (4, -1) >>>>> ? (5, 3) ? (8, -1) >>>>> row 6: (3, -1) ? (6, 2) >>>>> ? (7, -1) >>>>> row 7: (4, -1) ? (6, -1) >>>>> ? (7, 3) ? (8, -1) >>>>> row 8: (5, -1) ? (7, -1) >>>>> ? (8, 2) >>>>> Vector >>>>> Object:Vec_0000000084000000_0 >>>>> 1 MPI processes >>>>> ? type: mpi >>>>> Process [0] >>>>> 0.25 >>>>> 0.0205213 >>>>> 1.135e-005 >>>>> 0.0205213 >>>>> 0.00168449 >>>>> 9.31663e-007 >>>>> 1.135e-005 >>>>> 9.31663e-007 >>>>> 5.15289e-010 >>>>> Vector >>>>> Object:Vec_0000000084000000_1 >>>>> 1 MPI processes >>>>> ? type: mpi >>>>> Process [0] >>>>> 0.14924 >>>>> 0.0242397 >>>>> -0.0260347 >>>>> 0.0242397 >>>>> -0.0256192 >>>>> -0.0400102 >>>>> -0.0260347 >>>>> -0.0400102 >>>>> -0.0400102 >>>>> Press any key to continue >>>>> . . . >>>>> >>>>> C: >>>>> >>>>> Matrix Object: 1 MPI processes >>>>> ? type: seqaij >>>>> row 0: (0, 2) ? (1, -1) >>>>> ? (3, -1) >>>>> row 1: (0, -1) ? (1, 3) >>>>> ? (2, -1) ? (4, -1) >>>>> row 2: (1, -1) ? (2, 2) >>>>> ? (5, -1) >>>>> row 3: (0, -1) ? (3, 3) >>>>> ? (4, -1) ? (6, -1) >>>>> row 4: (1, -1) ? (3, -1) >>>>> ? (4, 4) ? (5, -1) ? (7, -1) >>>>> row 5: (2, -1) ? (4, -1) >>>>> ? (5, 3) ? (8, -1) >>>>> row 6: (3, -1) ? (6, 2) >>>>> ? (7, -1) >>>>> row 7: (4, -1) ? (6, -1) >>>>> ? (7, 3) ? (8, -1) >>>>> row 8: (5, -1) ? (7, -1) >>>>> ? (8, 2) >>>>> Vector >>>>> Object:Vec_0x1d3b000_0 1 >>>>> MPI processes >>>>> ? type: mpi >>>>> Process [0] >>>>> 0.25 >>>>> 0.0205212 >>>>> 1.135e-05 >>>>> 0.0205212 >>>>> 0.00168449 >>>>> 9.31663e-07 >>>>> 1.135e-05 >>>>> 9.31663e-07 >>>>> 5.15288e-10 >>>>> Vector >>>>> Object:Vec_0x1d3b000_1 1 >>>>> MPI processes >>>>> ? type: mpi >>>>> Process [0] >>>>> 0.139311 >>>>> 0.0305751 >>>>> -0.0220633 >>>>> 0.0305751 >>>>> -0.0135158 >>>>> -0.042185 >>>>> -0.0220633 >>>>> -0.042185 >>>>> -0.058449 >>>>> >>>>> >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >>>>> On 1/5/2012 11:54 PM, >>>>> Matthew Knepley wrote: >>>>> >>>>> On Tue, May 1, 2012 at >>>>> 5:48 PM, TAY >>>>> wee-beng>>>> > >>>>> ? wrote: >>>>> Hi, >>>>> >>>>> Do you mean my method >>>>> is wrong? >>>>> >>>>> I am following the >>>>> template of ex22f, >>>>> >>>>> where the variables >>>>> are declared as : >>>>> >>>>> PetscScalar ? v(5) >>>>> >>>>> MatStencil ? >>>>> row(4),col(4,5) >>>>> >>>>> Hence, >>>>> >>>>> for the neumann BC >>>>> >>>>> num = 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? if (j/=0) then >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? v(num) = >>>>> -rho*HxdHy >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_i,num) = >>>>> i >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_j,num) = >>>>> j-1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? num = num + 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? end if >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? if (i/=0) then >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? v(num) = >>>>> -rho*HydHx >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_i,num) = >>>>> i-1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_j,num) = >>>>> j >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? num = num + 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? end if >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? if (i/=mx-1) then >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? v(num) = >>>>> -rho*HydHx >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_i,num) = >>>>> i+1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_j,num) = >>>>> j >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? num = num + 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? end if >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? if (j/=my-1) then >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? v(num) = >>>>> -rho*HxdHy >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_i,num) = >>>>> i >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? >>>>> ? col(MatStencil_j,num) = >>>>> j+1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? ? ? num = num + 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? end if >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? v(num) = >>>>> ((num-1)/2.0)*rho*(HxdHy >>>>> + HydHx) >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? print *, v >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? col(MatStencil_i,num) = >>>>> i >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? col(MatStencil_j,num) = >>>>> j >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? !num = num + 1 >>>>> >>>>> ? ? ? ? ? ? ? >>>>> ? call >>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr) >>>>> >>>>> I do not get any more >>>>> out of range error. >>>>> However,my ans is >>>>> still different from >>>>> that of ex29 in C. >>>>> >>>>> This is very simple. >>>>> You have an error in >>>>> your code. Checking it >>>>> is very simple: run >>>>> the code and >>>>> break in >>>>> MatSetValues(). Make >>>>> sure ex29 makes calls >>>>> with exactly the same >>>>> indices as your ex29f. >>>>> >>>>> ? ? Matt >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> -- >>>>> What most >>>>> experimenters take for >>>>> granted before they >>>>> begin their >>>>> experiments is >>>>> infinitely more >>>>> interesting than any >>>>> results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters >>>>> take for granted before >>>>> they begin their >>>>> experiments is infinitely >>>>> more interesting than any >>>>> results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take >>>>> for granted before they begin >>>>> their experiments is >>>>> infinitely more interesting >>>>> than any results to which >>>>> their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for >>>>> granted before they begin their >>>>> experiments is infinitely more >>>>> interesting than any results to >>>>> which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for >>>>> granted before they begin their >>>>> experiments is infinitely more >>>>> interesting than any results to which >>>>> their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before >>>>> they begin their experiments is infinitely more >>>>> interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ! Concepts: KSP^solving a system of linear equations ! Concepts: KSP^Laplacian, 2d ! Processors: n !Added by Wee Beng Tay (W.B.Tay at tudelft.nl). !conversion from ex29 C version to Fortran !Inhomogeneous Laplacian in 2D. Modeled by the partial differential equation ! -div \rho grad u = f, 0 < x,y < 1, !with forcing function ! f = e^{-x^2/\nu} e^{-y^2/\nu} !with Dirichlet boundary conditions ! u = f(x,y) for x = 0, x = 1, y = 0, y = 1 !or pure Neumman boundary conditions !This uses multigrid to solve the linear system program ex29f implicit none ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Include files ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! ! petscsys.h - base PETSc routines petscvec.h - vectors ! petscmat.h - matrices ! petscksp.h - Krylov subspace methods petscpc.h - preconditioners #include #include #include #include #include #include #include !#include "finclude/petsc.h90" PetscErrorCode ierr DM da KSP ksp Vec x external ComputeRHS,ComputeMatrix PetscInt i1,i3,bc_cond !>bc_cond = boundary condition = 1/2 = dirichlet/neumann !>have to change bc_cond at 3 locations call PetscInitialize(PETSC_NULL_CHARACTER,ierr) bc_cond = 2 i3 = -3 i1 = 1 call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) call DMSetFunction(da,ComputeRHS,ierr) call DMSetJacobian(da,ComputeMatrix,ierr) call KSPSetDM(ksp,da,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSetUp(ksp,ierr) call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) call KSPGetSolution(ksp,x,ierr) call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) call KSPDestroy(ksp,ierr) call DMDestroy(da,ierr) call PetscFinalize(ierr) end program ex29f subroutine ComputeRHS(da,x,b,ierr) implicit none #include #include #include #include #include #include #include PetscErrorCode ierr PetscInt i,j,ij,mx,my,xm,ym,xs,ys,bc_cond,status PetscScalar h,nu,rho PetscScalar Hx,Hy !PetscScalar,pointer :: array(:,:) !Problem with using DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90 below in Vec x,b !linux with ifort, so VecSetValues was used. Worked in VS2008 though DM da MatNullSpace nullspace !>neumann BC integer, allocatable:: ij_array(:) real(8), allocatable :: array(:) nu = 0.1 bc_cond = 2 call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) Hx = 1.d0 / (mx-1) Hy = 1.d0 / (my-1) call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) allocate (ij_array(0:(ym-ys)*(xm-xs)-1), STAT=status) if (status/=0) STOP "Cannot allocate memory" allocate (array(0:(ym-ys)*(xm-xs)-1), STAT=status) if (status/=0) STOP "Cannot allocate memory" !call VecCreateMPI(PETSC_COMM_WORLD,PETSC_DECIDE,(ym-ys)*(xm-xs),b,ierr) !call DMDAVecGetArrayF90(da,b,array,ierr) !do j = ys,ys+ym-1 ! do i = xs,xs+xm-1 ! array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy ! end do !end do !call DMDAVecRestoreArrayF90(da,b,array,ierr) do j = ys,ys+ym-1 do i = xs,xs+xm-1 ij = (j)*(xm-xs) + i ij_array(ij) = ij array(ij) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy end do end do call VecSetValues(b,(ym-ys)*(xm-xs),ij_array,array,INSERT_VALUES,ierr) call VecAssemblyBegin(b,ierr) call VecAssemblyEnd(b,ierr) !call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr) if (bc_cond == 2) then call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) call MatNullSpaceRemove(nullspace,b,PETSC_NULL_OBJECT,ierr) call MatNullSpaceDestroy(nullspace,ierr) end if end subroutine ComputeRHS subroutine ComputeMatrix(da,x,JJ,jac,str,ierr) implicit none #include #include #include #include #include Mat jac,JJ PetscErrorCode ierr DM da PetscInt i,j,k,mx,my,xm,num PetscInt ym,xs,ys,i1,i5,bc_cond PetscScalar v(5),Hx,Hy,rho,centerRho PetscScalar HydHx PetscScalar HxdHy MatStencil row(4),col(4,5) Vec x MatStructure str MatNullSpace nullspace !>neumann BC bc_cond = 2 rho = 1.0 i1 = 1 i5 = 5 centerRho = rho call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) Hx = 1.d0 / (mx-1) Hy = 1.d0 / (my-1) HxdHy = Hx/Hy HydHx = Hy/Hx call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) do j=ys,ys+ym-1 do i=xs,xs+xm-1 row(MatStencil_i) = i row(MatStencil_j) = j call ComputeRho(i,j,mx,my,centerRho,rho) if (i==0 .or. j==0 .or. i==mx-1 .or. j==my-1) then if (bc_cond == 1) then v(1) = 2.0*rho*(HxdHy + HydHx) call MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr) else if (bc_cond == 2) then num = 1 if (j/=0) then v(num) = -rho*HxdHy col(MatStencil_i,num) = i col(MatStencil_j,num) = j-1 num = num + 1 end if if (i/=0) then v(num) = -rho*HydHx col(MatStencil_i,num) = i-1 col(MatStencil_j,num) = j num = num + 1 end if if (i/=mx-1) then v(num) = -rho*HydHx col(MatStencil_i,num) = i+1 col(MatStencil_j,num) = j num = num + 1 end if if (j/=my-1) then v(num) = -rho*HxdHy col(MatStencil_i,num) = i col(MatStencil_j,num) = j+1 num = num + 1 end if v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx) col(MatStencil_i,num) = i col(MatStencil_j,num) = j !num = num + 1 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr) end if else v(1) = -rho*HxdHy col(MatStencil_i,1) = i col(MatStencil_j,1) = j-1 v(2) = -rho*HydHx col(MatStencil_i,2) = i-1 col(MatStencil_j,2) = j v(3) = 2.0*rho*(HxdHy + HydHx) col(MatStencil_i,3) = i col(MatStencil_j,3) = j v(4) = -rho*HydHx col(MatStencil_i,4) = i+1 col(MatStencil_j,4) = j v(5) = -rho*HxdHy col(MatStencil_i,5) = i col(MatStencil_j,5) = j+1 call MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr) end if end do end do call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr) !call MatView(jac,PETSC_VIEWER_STDOUT_WORLD,ierr) if (bc_cond == 2) then call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) call MatSetNullSpace(jac,nullspace,ierr) call MatNullSpaceDestroy(nullspace,ierr) end if end subroutine ComputeMatrix subroutine ComputeRho(i,j,mx,my,centerRho,rho) PetscInt i,j,mx,my PetscScalar rho,centerRho if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j < 2.0*my/3.0)) then rho = centerRho else rho = 1.0 end if end subroutine ComputeRho From w_ang_temp at 163.com Sat Jun 2 07:02:43 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sat, 2 Jun 2012 20:02:43 +0800 (CST) Subject: [petsc-users] About the ./configure of petsc and mpi Message-ID: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> Hello, I reconfigure the mpi and petsc today. I want to use ifort as the compiler. And there is a problem. Before I reconfigure them, both compiling and running are ok (1.make ex4f; 2.mpiexec -n 2 ./ex4f;The name 'ex4f' is my project name, not the petsc example name).And I have done lots of work before.Since I think the project is inefficient, I want to use ifort. After the reconfiguration, the compiling seems ok(make ex4f) and I can get the executable file. But when I run it(mpiexec -n 2 ./ex4f), it does not work ('forrtl: severe (32): invalid logical unit number, unit -1215421032, file unknown'). Besides, both compiling and running are ok when I test the petsc examples. Before I reconfigure them, the configurations are as follows: mpi: ./configure --prefix=/home/geo/soft/mpich2 PETSc: ./configure --download-f-blas-lapack=1 And now the configurations are the following: mpi: ./configure --prefix=/home/geo/soft/mpich2 CC=icc CXX=icpc F77=ifort FC=ifort PETSc: ./configure --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 I do not know why it does not work when I run it after the reconfiguration. The information of the compiling is as follows. (1)Information(make ex4f) before the reconfiguration: mpif90 -c -I/home/ddc/soft/petsc/petsc-3.2-p7/include/finclude -g -I/home/ddc/soft/petsc/petsc-3.2-p7/include -I/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/include -I/home/ddc/soft/mpich2/include -o ex4f.o ex4f.F ---------------------------------^ mpif90 -g -I/home/ddc/soft/petsc/petsc-3.2-p7/include/finclude -o ex4f ex4f.o -L/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib -lpetsc -lpthread -Wl,-rpath,/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/home/ddc/soft/mpich2/lib -L/usr/lib/gcc/i486-linux-gnu/4.4.3 -L/home/ddc/intel/composer_xe_2011_sp1.9.293/compiler/lib/ia32 -L/home/ddc/intel/composer_xe_2011_sp1.9.293/mkl/lib/ia32 -L/usr/lib/i486-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl /bin/rm -f -f ex4f.o (2)Information(make ex4f) after the reconfiguration: /home/geo/soft/mpich2/bin/mpif90 -c -I/home/geo/soft/petsc/petsc-3.2-p7/include/finclude -g -I/home/geo/soft/petsc/petsc-3.2-p7/include -I/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/include -I/home/geo/soft/mpich2/include -o ex4f.o ex4f.F ---------------------------------^ /home/geo/soft/mpich2/bin/mpif90 -g -I/home/geo/soft/petsc/petsc-3.2-p7/include/finclude -o ex4f ex4f.o -L/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib -lpetsc -lpthread -Wl,-rpath,/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib -lflapack -lfblas -ldl -L/home/geo/soft/mpich2/lib -lmpich -lopa -lmpl -lrt -lpthread -L/opt/intel/composer_xe_2011_sp1.10.319/compiler/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/ipp/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/mkl/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.10.319/tbb/lib/ia32/cc4.1.0_libc2.4_kernel2.6.16.21 -L/opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/ia32 -L/opt/intel/composer_xe_2011_sp1.9.293/mkl/lib/ia32 -L/usr/lib/gcc/i486-linux-gnu/4.4.3 -L/usr/lib/i486-linux-gnu -limf -lsvml -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -lmpichf90 -lifport -lifcore -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -limf -lsvml -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -ldl /bin/rm -f -f ex4f.o (3)When reconfiguring the petsc, the information about compilers is as follows. Compilers: C Compiler: /home/geo/soft/mpich2/bin/mpicc -wd1572 -Qoption,cpp,--extended_float_type -g Fortran Compiler: /home/geo/soft/mpich2/bin/mpif90 -g Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 2 08:11:05 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 2 Jun 2012 08:11:05 -0500 Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> Message-ID: On Sat, Jun 2, 2012 at 7:02 AM, w_ang_temp wrote: > Hello, > I reconfigure the mpi and petsc today. I want to use ifort as the > compiler. > And there is a problem. > Before I reconfigure them, both compiling and running are ok (1.make > ex4f; > 2.mpiexec -n 2 ./ex4f;The name 'ex4f' is my project name, not the petsc > example name).And I have done lots of work before.Since I think the project > is inefficient, I want to use ifort. > After the reconfiguration, the compiling seems ok(make ex4f) and I can > get > the executable file. But when I run it(mpiexec -n 2 ./ex4f), it does not > work > ('forrtl: severe (32): invalid logical unit number, unit -1215421032, file > unknown'). > Show us the full stack trace (the full PETSc error message is good, a gdb stack trace would be better). It's quite possible that that you have some code that is not standards compliant, thus it behaves differently with different compilers. > Besides, both compiling and running are ok when I test the petsc examples. > Before I reconfigure them, the configurations are as follows: > mpi: ./configure --prefix=/home/geo/soft/mpich2 > PETSc: ./configure --download-f-blas-lapack=1 > And now the configurations are the following: > mpi: ./configure --prefix=/home/geo/soft/mpich2 > CC=icc CXX=icpc F77=ifort FC=ifort > PETSc: ./configure --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > I do not know why it does not work when I run it after the > reconfiguration. > > The information of the compiling is as follows. > (1)Information(make ex4f) before the reconfiguration: > mpif90 -c -I/home/ddc/soft/petsc/petsc-3.2-p7/include/finclude -g > -I/home/ddc/soft/petsc/petsc-3.2-p7/include > -I/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/include > -I/home/ddc/soft/mpich2/include > -o ex4f.o ex4f.F > ---------------------------------^ > mpif90 -g -I/home/ddc/soft/petsc/petsc-3.2-p7/include/finclude -o ex4f > ex4f.o > -L/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > -lpetsc -lpthread > -Wl,-rpath,/home/ddc/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > -lflapack -lfblas -lm > -L/home/ddc/soft/mpich2/lib -L/usr/lib/gcc/i486-linux-gnu/4.4.3 > -L/home/ddc/intel/composer_xe_2011_sp1.9.293/compiler/lib/ia32 > -L/home/ddc/intel/composer_xe_2011_sp1.9.293/mkl/lib/ia32 > -L/usr/lib/i486-linux-gnu -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s > -lmpichf90 -lifport > -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -ldl -lmpich -lopa > -lmpl -lrt > -lpthread -lgcc_s -ldl > /bin/rm -f -f ex4f.o > (2)Information(make ex4f) after the reconfiguration: > /home/geo/soft/mpich2/bin/mpif90 -c > -I/home/geo/soft/petsc/petsc-3.2-p7/include/finclude -g > -I/home/geo/soft/petsc/petsc-3.2-p7/include > -I/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/include > -I/home/geo/soft/mpich2/include > -o ex4f.o ex4f.F > ---------------------------------^ > /home/geo/soft/mpich2/bin/mpif90 -g > -I/home/geo/soft/petsc/petsc-3.2-p7/include/finclude -o ex4f ex4f.o > -L/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > -lpetsc -lpthread > -Wl,-rpath,/home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > -lflapack -lfblas -ldl > -L/home/geo/soft/mpich2/lib -lmpich -lopa -lmpl -lrt -lpthread > -L/opt/intel/composer_xe_2011_sp1.10.319/compiler/lib/ia32 > -L/opt/intel/composer_xe_2011_sp1.10.319/ipp/lib/ia32 > -L/opt/intel/composer_xe_2011_sp1. 10.319/mkl/lib/ia32 > -L/opt/intel/composer_xe_2011_sp1.10.319/tbb/lib/ia32/cc4.1.0_libc2.4_kernel2.6.16.21 > > -L/opt/intel/composer_xe_2011_sp1.9.293/compiler/lib/ia32 > -L/opt/intel/composer_xe_2011_sp1.9.293/mkl/lib/ia32 > -L/usr/lib/gcc/i486-linux-gnu/4.4.3 > -L/usr/lib/i486-linux-gnu -limf -lsvml -lipgo -ldecimal -lcilkrts > -lstdc++ -lgcc_s > -lirc -lirc_s -lmpichf90 -lifport -lifcore -lm -lm -ldl -lmpich -lopa > -lmpl -lrt > -lpthread -limf -lsvml -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc > -lirc_s -ldl > /bin/rm -f -f ex4f.o > (3)When reconfiguring the petsc, the information about compilers is as > follows. > Compilers: > C Compiler: /home/geo/soft/mpich2/bin/mpicc -wd1572 > -Qoption,cpp,--extended_float_type -g > Fortran Compiler: /home/geo/soft/mpich2/bin/mpif90 -g > Thanks. > Jim > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frtr at fysik.dtu.dk Fri Jun 1 13:21:38 2012 From: frtr at fysik.dtu.dk (Frederik Treue) Date: Fri, 01 Jun 2012 20:21:38 +0200 Subject: [petsc-users] How to use external vectors in a KSPmonitor In-Reply-To: References: <4FC8D5FC.2090203@risoe.dtu.dk> Message-ID: <4FC90832.6010601@fysik.dtu.dk> On 2012-06-01 16:52, Jed Brown wrote: > On Fri, Jun 1, 2012 at 9:47 AM, Frederik Treue > wrote: > > I need to access a vector from outside the ksp context in my own, > user-defined KSPmonitor. I have tried using the "context" input > variable in the KSPMonitor function to pass a pointer to the > relevant structure, but I get segmentation faults: Can this > "context" variable be used for this, and if no, how does one do? > > > This is what the context is for. Send the complete output from the > SEGV and any relevant snippets of code showing how you use it. nevermind, it turned out to be a trivial error on my part. /Frederik Treue -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sat Jun 2 09:11:18 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sat, 2 Jun 2012 22:11:18 +0800 (CST) Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> Message-ID: <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> Hello, Jed I use gdb to debug it. ?1?The source code(The line number is add by me.Line 125 and 127 are added right now for debugging ): ............ 121 open(50,FILE='check.out') 122 open(LSS,FILE='select.dat') 123 C open(LH,FILE='select1.dat',FORM='UNFORMATTED') 124 open(LR,FILE='initials.dat') 125 write(*,*) 'test1' 126 open(LTR,FILE='trapdoor.dat') 127 write(*,*) 'test2' ?2? Debug information: (gdb) break 124 Breakpoint 1 at 0x804bb42: file ex4f.F, line 124. (gdb) run Starting program: /home/geo/project/temp/8-little-9-2716/tutorials/ex4f [Thread debugging using libthread_db enabled] Breakpoint 1, gleaves () at ex4f.F:124 124 open(LR,FILE='initials.dat') (gdb) n 125 write(*,*) 'test1' (gdb) n test1 126 open(LTR,FILE='trapdoor.dat') (gdb) n forrtl: severe (32): invalid logical unit number, unit -1208035944, file unknown Image PC Routine Line Source libirc.so 00AC3470 Unknown Unknown Unknown libirc.so 00AC2011 Unknown Unknown Unknown libifcore.so.5 00BD066B Unknown Unknown Unknown libifcore.so.5 00B5A178 Unknown Unknown Unknown libifcore.so.5 00B70CF4 Unknown Unknown Unknown ex4f 0804BC6B Unknown Unknown Unknown ex4f 0804B904 Unknown Unknown Unknown libc.so.6 00C48BD6 Unknown Unknown Unknown ex4f 0804B811 Unknown Unknown Unknown Program exited with code 040. (gdb) ?3?From the information above, it stops in 126 and can not go to line 127. Thanks. Jim >? 2012-06-02 21:11:05?"Jed Brown" ??? >On Sat, Jun 2, 2012 at 7:02 AM, w_ang_temp wrote: >Hello, > I reconfigure the mpi and petsc today. I want to use ifort as the compiler. >And there is a problem. > Before I reconfigure them, both compiling and running are ok (1.make ex4f; >2.mpiexec -n 2 ./ex4f;The name 'ex4f' is my project name, not the petsc >example name).And I have done lots of work before.Since I think the project >is inefficient, I want to use ifort. > After the reconfiguration, the compiling seems ok(make ex4f) and I can get >the executable file. But when I run it(mpiexec -n 2 ./ex4f), it does not work >('forrtl: severe (32): invalid logical unit number, unit -1215421032, file unknown'). >Show us the full stack trace (the full PETSc error message is good, a gdb stack trace would be better). It's quite possible that that you have >some code that is not standards compliant, thus it behaves differently with different compilers. >Besides, both compiling and running are ok when I test the petsc examples. > Before I reconfigure them, the configurations are as follows: > mpi: ./configure --prefix=/home/geo/soft/mpich2 > PETSc: ./configure --download-f-blas-lapack=1 > And now the configurations are the following: -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 2 09:13:06 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 2 Jun 2012 09:13:06 -0500 Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> Message-ID: Looks like LTR is uninitialized or has been overwritten by earlier corruption. This does not have anything to do with PETSc. On Sat, Jun 2, 2012 at 9:11 AM, w_ang_temp wrote: > Hello, Jed > I use gdb to debug it. > ?1?The source code(The line number is add by me.Line 125 and 127 are > added right now for debugging ): > ............ > 121 open(50,FILE='check.out') > 122 open(LSS,FILE='select.dat') > 123 C open(LH,FILE='select1.dat',FORM='UNFORMATTED') > 124 open(LR,FILE='initials.dat') > 125 write(*,*) 'test1' > 126 open(LTR,FILE='trapdoor.dat') > 127 write(*,*) 'test2' > ?2? Debug information: > (gdb) break 124 > Breakpoint 1 at 0x804bb42: file ex4f.F, line 124. > (gdb) run > Starting program: /home/geo/project/temp/8-little-9-2716/tutorials/ex4f > [Thread debugging using libthread_db enabled] > Breakpoint 1, gleaves () at ex4f.F:124 > 124 open(LR,FILE='initials.dat') > (gdb) n > 125 write(*,*) 'test1' > (gdb) n > test1 > 126 open(LTR,FILE='trapdoor.dat') > (gdb) n > forrtl: severe (32): invalid logical unit number, unit -1208035944, file > unknown > Image PC Routine Line > Source > libirc.so 00AC3470 Unknown Unknown Unknown > libirc.so 00AC2011 Unknown Unknown Unknown > libifcore.so.5 00BD066B Unknown Unknown Unknown > libifcore.so.5 00B5A178 Unknown Unknown Unknown > libifcore.so.5 00B70CF4 Unknown Unknown Unknown > ex4f 0804BC6B Unknown Unknown Unknown > ex4f 0804B904 Unknown Unknown Unknown > libc.so.6 00C48BD6 Unknown Unknown Unknown > ex4f 0804B811 Unknown Unknown Unknown > Program exited with code 040. > (gdb) > ?3?From the information above, it stops in 126 and can not go to line > 127. > > Thanks. > Jim > > >? 2012-06-02 21:11:05?"Jed Brown" ??? > > >On Sat, Jun 2, 2012 at 7:02 AM, w_ang_temp wrote: > >> >Hello, >> > I reconfigure the mpi and petsc today. I want to use ifort as the >> compiler. >> >And there is a problem. >> > Before I reconfigure them, both compiling and running are ok (1.make >> ex4f; >> >2.mpiexec -n 2 ./ex4f;The name 'ex4f' is my project name, not the petsc >> >example name).And I have done lots of work before.Since I think the >> project >> >is inefficient, I want to use ifort. >> > After the reconfiguration, the compiling seems ok(make ex4f) and I >> can get >> >the executable file. But when I run it(mpiexec -n 2 ./ex4f), it does not >> work >> >('forrtl: severe (32): invalid logical unit number, unit -1215421032, >> file unknown'). >> > > >Show us the full stack trace (the full PETSc error message is good, a gdb > stack trace would be better). It's quite possible that that you have >some > code that is not standards compliant, thus it behaves differently with > different compilers. > > >> >Besides, both compiling and running are ok when I test the petsc >> examples. >> > Before I reconfigure them, the configurations are as follows: >> > mpi: ./configure --prefix=/home/geo/soft/mpich2 >> > PETSc: ./configure --download-f-blas-lapack=1 >> > And now the configurations are the following: >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sat Jun 2 09:28:36 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sat, 2 Jun 2012 22:28:36 +0800 (CST) Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> Message-ID: <36a9551f.1c59b.137ad9909f1.Coremail.w_ang_temp@163.com> Hello Jed It is true that LTR is uninitialized. But before the reconfiguration, it is ok by the default compiler. The codes are the same before and after the reconfiguration. Also I use ifort to compile the fortran project before I add petsc code to it and it is ok. So I think that this line (line 126) should not be incorrect when use ifort. Maybethere'sany otherreason related to the new compiler. Thanks. Jim >At 2012-06-02 22:13:06,"Jed Brown" wrote: >Looks like LTR is uninitialized or has been overwritten by earlier corruption. This does not have anything to do with PETSc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 2 09:34:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 2 Jun 2012 09:34:58 -0500 Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: <36a9551f.1c59b.137ad9909f1.Coremail.w_ang_temp@163.com> References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> <36a9551f.1c59b.137ad9909f1.Coremail.w_ang_temp@163.com> Message-ID: On Sat, Jun 2, 2012 at 9:28 AM, w_ang_temp wrote: > Hello Jed > It is true that LTR is uninitialized. > There are lots of ways for uninitialized variables to accidentally have valid values. That doesn't mean your program is correct, it just means that it accidentally "worked". You have to fix your code. http://fortran90.org/src/best-practices.html#file-input-output > But before the reconfiguration, it is ok by the default compiler. The > codes are the same before and after the reconfiguration. > Also I use ifort to compile the fortran project before I add petsc > code to it and it is ok. So I think that this line (line 126) should not be > incorrect when use ifort. > Maybe there's any other reason related to the new compiler. > Thanks. > Jim > >At 2012-06-02 22:13:06,"Jed Brown" wrote: > > >Looks like LTR is uninitialized or has been overwritten by earlier > corruption. This does not have anything to do with PETSc. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sat Jun 2 09:37:20 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sat, 2 Jun 2012 22:37:20 +0800 (CST) Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> <36a9551f.1c59b.137ad9909f1.Coremail.w_ang_temp@163.com> Message-ID: <2f6d12e6.1a397.137ada10595.Coremail.w_ang_temp@163.com> OK. Thank you very much! >? 2012-06-02 22:34:58?"Jed Brown" ??? >On Sat, Jun 2, 2012 at 9:28 AM, w_ang_temp wrote: >Hello Jed > It is true that LTR is uninitialized. >There are lots of ways for uninitialized variables to accidentally have valid values. That doesn't mean your program is correct, it just means that it >accidentally "worked". You have to fix your code. >http://fortran90.org/src/best-practices.html#file-input-output -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiangwen84 at gmail.com Sat Jun 2 10:47:21 2012 From: jiangwen84 at gmail.com (Wen Jiang) Date: Sat, 2 Jun 2012 11:47:21 -0400 Subject: [petsc-users] matload big matrix Message-ID: Hi, I am trying to load a big matrix(size 552147 by 552147) from a binary file(around 2GB). But I found my code seems to get stuck at the loading stage. I copied the last a few info output below [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 [0] PetscFileRetrieve(): Found file l2projphim.dat [0] PetscFileRetrieve(): Found file l2projphim.dat.info [0] PetscOptionsInsertFile(): Opened options file l2projphim.dat.info Do you have any suggestions? Thanks. Regards, Wen -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 2 10:53:47 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 2 Jun 2012 10:53:47 -0500 Subject: [petsc-users] matload big matrix In-Reply-To: References: Message-ID: How long is it taking? What file system are you reading the matrix from? You can attach a debugger and see where it's "stuck". On Sat, Jun 2, 2012 at 10:47 AM, Wen Jiang wrote: > Hi, > > I am trying to load a big matrix(size 552147 by 552147) from a binary > file(around 2GB). But I found my code seems to get stuck at the loading > stage. I copied the last a few info output below > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 > -2080374780 > [0] PetscFileRetrieve(): Found file l2projphim.dat > [0] PetscFileRetrieve(): Found file l2projphim.dat.info > [0] PetscOptionsInsertFile(): Opened options file l2projphim.dat.info > > Do you have any suggestions? Thanks. > > Regards, > Wen > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jun 2 10:55:50 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Jun 2012 10:55:50 -0500 Subject: [petsc-users] matload big matrix In-Reply-To: References: Message-ID: What version of PETSc is this? How was the file generated? How many processes are you using? Does it work on less processes (like 1, 2 )? How many nonzeros in the matrix (roughly)? Barry > Hi, > > I am trying to load a big matrix(size 552147 by 552147) from a binary file(around 2GB). But I found my code seems to get stuck at the loading stage. I copied the last a few info output below > > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374780 > [0] PetscFileRetrieve(): Found file l2projphim.dat > [0] PetscFileRetrieve(): Found file l2projphim.dat.info > [0] PetscOptionsInsertFile(): Opened options file l2projphim.dat.info > > Do you have any suggestions? Thanks. > > Regards, > Wen > > From jiangwen84 at gmail.com Sat Jun 2 12:31:46 2012 From: jiangwen84 at gmail.com (Wen Jiang) Date: Sat, 2 Jun 2012 13:31:46 -0400 Subject: [petsc-users] matload big matrix Message-ID: Thanks for your reply. How long is it taking? What file system are you reading the matrix from? It seems to never finish. Sorry, I do not know what does the file system mean. What version of PETSc is this? How was the file generated? How many processes are you using? Does it work on less processes (like 1, 2 )? How many nonzeros in the matrix (roughly)? I am using 64 processes. The nonzeros in the matrix is around 8 billion. I am using PETSc dev. But the binary file was generated by PETSc 3.2 PetscViewerBinaryOpen(PETSC_COMM_WORLD,"l2projphim.dat",FILE_MODE_WRITE,&viewer). I just tried to use PETSc 3.2 to read and it had no problem. For a smaller matrix, PETSc dev successfully read the binary file generated by PETSc 3.2. But I do not know why it does not work for large matrix. Thanks, Regards, Wen -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Jun 2 13:17:31 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 2 Jun 2012 13:17:31 -0500 Subject: [petsc-users] matload big matrix In-Reply-To: References: Message-ID: <22606948-A5C9-4859-B130-083CF3DB660D@mcs.anl.gov> On Jun 2, 2012, at 12:31 PM, Wen Jiang wrote: > Thanks for your reply. > > How long is it taking? What file system are you reading the matrix from? > > It seems to never finish. Sorry, I do not know what does the file system mean. > > What version of PETSc is this? How was the file generated? How many processes are you using? Does it work on less processes (like 1, 2 )? How many nonzeros in the matrix (roughly)? > > I am using 64 processes. The nonzeros in the matrix is around 8 billion. > > I am using PETSc dev. But the binary file was generated by PETSc 3.2 PetscViewerBinaryOpen(PETSC_COMM_WORLD,"l2projphim.dat",FILE_MODE_WRITE,&viewer). I just tried to use PETSc 3.2 to read and it had no problem. For a smaller matrix, PETSc dev successfully read the binary file generated by PETSc 3.2. But I do not know why it does not work for large matrix. > Are both PETSc's configured with --with-64-bit-indices? Can you read it with fewer processes? Like 32 or 16? barry > Thanks, > > Regards, > Wen > > > > > From bojan.niceno at psi.ch Sun Jun 3 05:00:47 2012 From: bojan.niceno at psi.ch (Bojan Niceno) Date: Sun, 03 Jun 2012 12:00:47 +0200 Subject: [petsc-users] Solutions different after program restart Message-ID: <4FCB35CF.6040505@psi.ch> Dear all, I have a rather strange issue with the incompressible Navier-Stokes solver I am developing. It works fine with PETSc, unless I do a /restart/. By restart I mean loading the values of dependent variables from previously performed simulation. After such a restart, results obtained from PETSc linear solver are slightly different than those obtained from the previous simulation. For example, I make a first run with 100 time steps, save dependent values at 50. I run another simulation, which starts from the results saved at 50, but the results in the first time step in the new run will be slightly different than those of step 51 in the previous run. The differences are small, but still give an uneasy feeling. I suspect the difference /must /stem from different values of: 1 -dependent variables (x), or 2 - sources (b), or 3 - system matrices (A). In order to pinpoint the cause of the differences, I print norms of A, x, b before and after restart. They are exactly the same which, to my understanding, should ensure the results from a call to linear solver are the same. Yet, that doesn't seem to be the case. Can anybody give me an advice on how to find the cause of these differences? Is there something inside PETSc objects (such as helping vectors or alike) which might need cleaning in order to ensure exactly the same results after the program restart? Kind regards, Bojan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiangwen84 at gmail.com Sun Jun 3 07:44:39 2012 From: jiangwen84 at gmail.com (Wen Jiang) Date: Sun, 3 Jun 2012 08:44:39 -0400 Subject: [petsc-users] petsc dev mat assembly fails Message-ID: Hi, I switched my code from petsc 3.2 to petsc dev in order to use the newest superlu dist. And I also change MatCreateMPIAIJ to MatCreateAIJ as mentioned on developer site. However I got some errors which I did not have in petsc 3.2. From tracking the -info output, I found my code did not execute the MatAssembly correctly. The output file is attached. Thus I am wondering what else I should pay attention to when switching from 3.2 to dev. Thanks. Regards, Wen -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_info Type: application/octet-stream Size: 101538 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Jun 3 08:15:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 08:15:22 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <4FCB35CF.6040505@psi.ch> References: <4FCB35CF.6040505@psi.ch> Message-ID: How large are the differences? If they are at linear (or nonlinear) solver tolerance, then both are equally correct (or equally wrong). For some codes, the domain decomposition will not be identical on restart; that changes the algorithm. Some preconditioners actually use randomness in their setup (e.g. ML). VecScatter processes messages in the order that they are received (best for performance); you can use -vecscatter_reproduce (or one of the other -vecscatter options) to process messages in deterministic order. The matrix stash (used to assemble off-process entries, usually with ADD_VALUES) does not currently have an option to reproduce exactly so finite element assembly can vary (at rounding error). How important is bitwise reproducibility (understanding that it's expensive and limits the available algorithms) versus reproducibility to any desired tolerance? On Sun, Jun 3, 2012 at 5:00 AM, Bojan Niceno wrote: > Dear all, > > > I have a rather strange issue with the incompressible Navier-Stokes solver > I am developing. It works fine with PETSc, unless I do a *restart*. > > By restart I mean loading the values of dependent variables from > previously performed simulation. After such a restart, results obtained > from PETSc linear solver are slightly different than those obtained from > the previous simulation. > > For example, I make a first run with 100 time steps, save dependent values > at 50. > > I run another simulation, which starts from the results saved at 50, but > the results in the first time step in the new run will be slightly > different than those of step 51 in the previous run. The differences are > small, but still give an uneasy feeling. > > I suspect the difference *must *stem from different values of: > 1 -dependent variables (x), or > 2 - sources (b), or > 3 - system matrices (A). > > In order to pinpoint the cause of the differences, I print norms of A, x, > b before and after restart. They are exactly the same which, to my > understanding, should ensure the results from a call to linear solver are > the same. Yet, that doesn't seem to be the case. > > Can anybody give me an advice on how to find the cause of these > differences? > > Is there something inside PETSc objects (such as helping vectors or alike) > which might need cleaning in order to ensure exactly the same results after > the program restart? > > > > Kind regards, > > > Bojan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 3 08:19:08 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 08:19:08 -0500 Subject: [petsc-users] petsc dev mat assembly fails In-Reply-To: References: Message-ID: On Sun, Jun 3, 2012 at 7:44 AM, Wen Jiang wrote: > Hi, > > I switched my code from petsc 3.2 to petsc dev in order to use the newest > superlu dist. And I also change MatCreateMPIAIJ to MatCreateAIJ as > mentioned on developer site. However I got some errors which I did not have > in petsc 3.2. From tracking the -info output, I found my code did not > execute the MatAssembly correctly. The output file is attached. Thus I am > wondering what else I should pay attention to when switching from 3.2 to > dev. Thanks. > There aren't any errors in this -info output? What did you see? Send the whole error message. > Regards, > Wen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bojan.niceno at psi.ch Sun Jun 3 08:30:53 2012 From: bojan.niceno at psi.ch (Niceno Bojan) Date: Sun, 3 Jun 2012 15:30:53 +0200 Subject: [petsc-users] Solutions different after program restart References: <4FCB35CF.6040505@psi.ch> Message-ID: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> Dear Jed, thanks for the answer. > How large are the differences? If they are at linear (or nonlinear) solver > tolerance, then both are equally correct (or equally wrong). Good point! For example: Residual norm obtained from a call to "KSPGetResidualNorm(ksp, &rnorm);", gives: - First run: 1.98077e-02 - Second run: 2.61329e-02 The tolerances I request from KSP's gmres solver are: - Linear solver relative tolerance = 0.001; - Linear solver absolute tolerance = 1.0e-5; Does it tell you something? It does not to me :-( > For some codes, the domain decomposition will not be identical on restart; > that changes the algorithm. That is clear, but what I am comparing are two sequential runs. Kind regards, Bojan Some preconditioners actually use randomness in their setup (e.g. ML). VecScatter processes messages in the order that they are received (best for performance); you can use -vecscatter_reproduce (or one of the other -vecscatter options) to process messages in deterministic order. The matrix stash (used to assemble off-process entries, usually with ADD_VALUES) does not currently have an option to reproduce exactly so finite element assembly can vary (at rounding error). How important is bitwise reproducibility (understanding that it's expensive and limits the available algorithms) versus reproducibility to any desired tolerance? On Sun, Jun 3, 2012 at 5:00 AM, Bojan Niceno wrote: > Dear all, > > > I have a rather strange issue with the incompressible Navier-Stokes solver > I am developing. It works fine with PETSc, unless I do a *restart*. > > By restart I mean loading the values of dependent variables from > previously performed simulation. After such a restart, results obtained > from PETSc linear solver are slightly different than those obtained from > the previous simulation. > > For example, I make a first run with 100 time steps, save dependent values > at 50. > > I run another simulation, which starts from the results saved at 50, but > the results in the first time step in the new run will be slightly > different than those of step 51 in the previous run. The differences are > small, but still give an uneasy feeling. > > I suspect the difference *must *stem from different values of: > 1 -dependent variables (x), or > 2 - sources (b), or > 3 - system matrices (A). > > In order to pinpoint the cause of the differences, I print norms of A, x, > b before and after restart. They are exactly the same which, to my > understanding, should ensure the results from a call to linear solver are > the same. Yet, that doesn't seem to be the case. > > Can anybody give me an advice on how to find the cause of these > differences? > > Is there something inside PETSc objects (such as helping vectors or alike) > which might need cleaning in order to ensure exactly the same results after > the program restart? > > > > Kind regards, > > > Bojan > From jedbrown at mcs.anl.gov Sun Jun 3 08:34:49 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 08:34:49 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> Message-ID: On Sun, Jun 3, 2012 at 8:30 AM, Niceno Bojan wrote: > Residual norm obtained from a call to "KSPGetResidualNorm(ksp, &rnorm);", > gives: > > - First run: 1.98077e-02 > - Second run: 2.61329e-02 > > The tolerances I request from KSP's gmres solver are: > > - Linear solver relative tolerance = 0.001; > - Linear solver absolute tolerance = 1.0e-5; > > Does it tell you something? It does not to me :-( > Is the initial residual about 30? > > For some codes, the domain decomposition will not be identical on > restart; > > that changes the algorithm. > > That is clear, but what I am comparing are two sequential runs. > In that case, check that the system being solved is identical. You can save it with -ksp_view_binary. Load it with src/ksp/ksp/examples/tutorials/ex10.c as an independent test. Also be sure to use the same initial guess. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bojan.niceno at psi.ch Sun Jun 3 08:37:36 2012 From: bojan.niceno at psi.ch (Niceno Bojan) Date: Sun, 3 Jun 2012 15:37:36 +0200 Subject: [petsc-users] Solutions different after program restart References: <4FCB35CF.6040505@psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> Message-ID: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858597@MAILBOX0B.psi.ch> Dear Jed, > > Residual norm obtained from a call to "KSPGetResidualNorm(ksp, &rnorm);", > > gives: > > > > - First run: 1.98077e-02 > > - Second run: 2.61329e-02 > > > > The tolerances I request from KSP's gmres solver are: > > > > - Linear solver relative tolerance = 0.001; > > - Linear solver absolute tolerance = 1.0e-5; > Is the initial residual about 30? I don't know. Is there a way to find out? > In that case, check that the system being solved is identical. You can save > it with -ksp_view_binary. Load it with > src/ksp/ksp/examples/tutorials/ex10.c as an independent test. Also be sure > to use the same initial guess. Thanks. I was checking norms of matrix "A" and "x" and "b" vectors (2nd order) and they ARE the same before and after restart. Does the same norm ensure the matrices and and vectors are the same? -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3120 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Jun 3 08:44:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 08:44:23 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858597@MAILBOX0B.psi.ch> References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858597@MAILBOX0B.psi.ch> Message-ID: On Sun, Jun 3, 2012 at 8:37 AM, Niceno Bojan wrote: > I don't know. Is there a way to find out? > -ksp_monitor or -ksp_monitor_true_residual > > > > In that case, check that the system being solved is identical. You can > save > > it with -ksp_view_binary. Load it with > > src/ksp/ksp/examples/tutorials/ex10.c as an independent test. Also be > sure > > to use the same initial guess. > > Thanks. I was checking norms of matrix "A" and "x" and "b" vectors (2nd > order) and they ARE the same before and after restart. Does the same norm > ensure the matrices and and vectors are the same? > They could be ordered differently. A different ordering would change the ILU algorithm. You can also run with -pc_type jacobi to avoid this source of variation (norms are still order-dependent up to rounding error). -------------- next part -------------- An HTML attachment was scrubbed... URL: From bojan.niceno at psi.ch Sun Jun 3 09:09:56 2012 From: bojan.niceno at psi.ch (Niceno Bojan) Date: Sun, 3 Jun 2012 16:09:56 +0200 Subject: [petsc-users] Solutions different after program restart References: <4FCB35CF.6040505@psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> Message-ID: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> Dear Jed, > > - First run: 1.98077e-02 > > - Second run: 2.61329e-02 > > > > The tolerances I request from KSP's gmres solver are: > > > > - Linear solver relative tolerance = 0.001; > > - Linear solver absolute tolerance = 1.0e-5; > Is the initial residual about 30? Well, even above that, around 50 :-) You were right. Reducing 50 by a factor of 0.001, which I set, the differences are indeed smaller than the solver tolerance :-) But, allow me to ask one more time, to make sure: I am using gmres with Jacobi's preconditioner. The run is sequential, the nodes are not renumbered. Is it safe to neglect these differences, even if they are smaller than prescribed tolerances? Kind regards, Bojan > > For some codes, the domain decomposition will not be identical on > restart; > > that changes the algorithm. > > That is clear, but what I am comparing are two sequential runs. > In that case, check that the system being solved is identical. You can save it with -ksp_view_binary. Load it with src/ksp/ksp/examples/tutorials/ex10.c as an independent test. Also be sure to use the same initial guess. From jedbrown at mcs.anl.gov Sun Jun 3 09:13:50 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 09:13:50 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> Message-ID: On Sun, Jun 3, 2012 at 9:09 AM, Niceno Bojan wrote: > Well, even above that, around 50 :-) > > You were right. Reducing 50 by a factor of 0.001, which I set, the > differences are indeed smaller than the solver tolerance :-) > Good > > > But, allow me to ask one more time, to make sure: I am using gmres with > Jacobi's preconditioner. The run is sequential, the nodes are not > renumbered. Is it safe to neglect these differences, even if they are > smaller than prescribed tolerances? > It's "safe" in the sense that both answers are equivalent, but it would be good to know what is causing the differences. How is the matrix assembled? Did you use -ksp_view_binary and ksp ex10 to check that the system really is identical? Does ksp ex10 give the same output each time you run it? Can you send a few iterations of output from -ksp_monitor_true_residual? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bojan.niceno at psi.ch Sun Jun 3 09:19:45 2012 From: bojan.niceno at psi.ch (Niceno Bojan) Date: Sun, 3 Jun 2012 16:19:45 +0200 Subject: [petsc-users] Solutions different after program restart References: <4FCB35CF.6040505@psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> Message-ID: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> > It's "safe" in the sense that both answers are equivalent, but it would be > good to know what is causing the differences. ... I agree. > How is the matrix assembled? I use a command like this: MatSetValues(A, 1, &gi, 1, &gj, &M[i][j], INSERT_VALUES); Always inserting values, rather than adding. > Did you use -ksp_view_binary and ksp ex10 to check that the system really > is identical? Does ksp ex10 give the same output each time you run it? Not yet. I am not that prudent with PETSc, I am currently celebrating for being able to see residual history :-) > Can you send a few iterations of output from -ksp_monitor_true_residual? Yes, they look like: 0 KSP preconditioned resid norm 5.321355215818e+01 true resid norm 1.650035042618e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.013755144242e+01 true resid norm 1.361285352452e+01 ||r(i)||/||b|| 8.250039043366e-01 2 KSP preconditioned resid norm 7.629376453731e+00 true resid norm 3.184316336783e-01 ||r(i)||/||b|| 1.929847702951e-02 3 KSP preconditioned resid norm 3.869976235279e+00 true resid norm 4.559806478792e-01 ||r(i)||/||b|| 2.763460387822e-02 4 KSP preconditioned resid norm 1.889666147602e+00 true resid norm 1.422513113059e-01 ||r(i)||/||b|| 8.621108499622e-03 5 KSP preconditioned resid norm 9.253616017337e-01 true resid norm 1.065191369186e-02 ||r(i)||/||b|| 6.455568164761e-04 6 KSP preconditioned resid norm 4.759923245260e-01 true resid norm 2.662360840276e-02 ||r(i)||/||b|| 1.613517756600e-03 7 KSP preconditioned resid norm 2.492423883098e-01 true resid norm 1.216652507263e-02 ||r(i)||/||b|| 7.373494961255e-04 8 KSP preconditioned resid norm 1.306776196573e-01 true resid norm 1.933734977205e-03 ||r(i)||/||b|| 1.171935702734e-04 9 KSP preconditioned resid norm 7.267055499898e-02 true resid norm 9.770926706361e-04 ||r(i)||/||b|| 5.921647997767e-05 10 KSP preconditioned resid norm 3.833602982459e-02 true resid norm 9.695036607433e-04 ||r(i)||/||b|| 5.875654975214e-05 velocity[0] took 10 iterations to reach residual 3.83360e-02 Kind regards, Bojan -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3936 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Jun 3 10:10:01 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 10:10:01 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> Message-ID: On Sun, Jun 3, 2012 at 9:19 AM, Niceno Bojan wrote: > I use a command like this: > > MatSetValues(A, 1, &gi, 1, &gj, &M[i][j], INSERT_VALUES); > > Always inserting values, rather than adding. > > > > Did you use -ksp_view_binary and ksp ex10 to check that the system really > > is identical? Does ksp ex10 give the same output each time you run it? > > Not yet. I am not that prudent with PETSc, I am currently celebrating for > being able to see residual history :-) > Just go build that example, then run it with -f path/to/binaryoutput. It will load the matrix and vector from the file and solve. > > Can you send a few iterations of output from -ksp_monitor_true_residual? > > Yes, they look like: > > 0 KSP preconditioned resid norm 5.321355215818e+01 true resid norm > 1.650035042618e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.013755144242e+01 true resid norm > 1.361285352452e+01 ||r(i)||/||b|| 8.250039043366e-01 > 2 KSP preconditioned resid norm 7.629376453731e+00 true resid norm > 3.184316336783e-01 ||r(i)||/||b|| 1.929847702951e-02 > 3 KSP preconditioned resid norm 3.869976235279e+00 true resid norm > 4.559806478792e-01 ||r(i)||/||b|| 2.763460387822e-02 > 4 KSP preconditioned resid norm 1.889666147602e+00 true resid norm > 1.422513113059e-01 ||r(i)||/||b|| 8.621108499622e-03 > 5 KSP preconditioned resid norm 9.253616017337e-01 true resid norm > 1.065191369186e-02 ||r(i)||/||b|| 6.455568164761e-04 > 6 KSP preconditioned resid norm 4.759923245260e-01 true resid norm > 2.662360840276e-02 ||r(i)||/||b|| 1.613517756600e-03 > 7 KSP preconditioned resid norm 2.492423883098e-01 true resid norm > 1.216652507263e-02 ||r(i)||/||b|| 7.373494961255e-04 > 8 KSP preconditioned resid norm 1.306776196573e-01 true resid norm > 1.933734977205e-03 ||r(i)||/||b|| 1.171935702734e-04 > 9 KSP preconditioned resid norm 7.267055499898e-02 true resid norm > 9.770926706361e-04 ||r(i)||/||b|| 5.921647997767e-05 > 10 KSP preconditioned resid norm 3.833602982459e-02 true resid norm > 9.695036607433e-04 ||r(i)||/||b|| 5.875654975214e-05 > velocity[0] took 10 iterations to reach residual 3.83360e-02 > Interesting that the true residual is dropping faster than preconditioned residual. Can you show the output from both runs? Are each individually reproducible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bojan.niceno at psi.ch Sun Jun 3 10:20:15 2012 From: bojan.niceno at psi.ch (Niceno Bojan) Date: Sun, 3 Jun 2012 17:20:15 +0200 Subject: [petsc-users] Solutions different after program restart References: <4FCB35CF.6040505@psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch><0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> Message-ID: <0C9CAFE8C4E5CA49884E636AE1D7CD690585859A@MAILBOX0B.psi.ch> Dear Jed, I will need some time with -ksp_view_binary. The file being created (called "binaryoutput") seems to grow as simulation time steps go by. It seems to me that all systems solved during a run are stored in the same file. I am struggling to understand how to pick the one I want to reproduce in ex10. Kind regards, Bojan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov on behalf of Jed Brown Sent: Sun 6/3/2012 5:10 PM To: PETSc users list Subject: Re: [petsc-users] Solutions different after program restart On Sun, Jun 3, 2012 at 9:19 AM, Niceno Bojan wrote: > I use a command like this: > > MatSetValues(A, 1, &gi, 1, &gj, &M[i][j], INSERT_VALUES); > > Always inserting values, rather than adding. > > > > Did you use -ksp_view_binary and ksp ex10 to check that the system really > > is identical? Does ksp ex10 give the same output each time you run it? > > Not yet. I am not that prudent with PETSc, I am currently celebrating for > being able to see residual history :-) > Just go build that example, then run it with -f path/to/binaryoutput. It will load the matrix and vector from the file and solve. > > Can you send a few iterations of output from -ksp_monitor_true_residual? > > Yes, they look like: > > 0 KSP preconditioned resid norm 5.321355215818e+01 true resid norm > 1.650035042618e+01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.013755144242e+01 true resid norm > 1.361285352452e+01 ||r(i)||/||b|| 8.250039043366e-01 > 2 KSP preconditioned resid norm 7.629376453731e+00 true resid norm > 3.184316336783e-01 ||r(i)||/||b|| 1.929847702951e-02 > 3 KSP preconditioned resid norm 3.869976235279e+00 true resid norm > 4.559806478792e-01 ||r(i)||/||b|| 2.763460387822e-02 > 4 KSP preconditioned resid norm 1.889666147602e+00 true resid norm > 1.422513113059e-01 ||r(i)||/||b|| 8.621108499622e-03 > 5 KSP preconditioned resid norm 9.253616017337e-01 true resid norm > 1.065191369186e-02 ||r(i)||/||b|| 6.455568164761e-04 > 6 KSP preconditioned resid norm 4.759923245260e-01 true resid norm > 2.662360840276e-02 ||r(i)||/||b|| 1.613517756600e-03 > 7 KSP preconditioned resid norm 2.492423883098e-01 true resid norm > 1.216652507263e-02 ||r(i)||/||b|| 7.373494961255e-04 > 8 KSP preconditioned resid norm 1.306776196573e-01 true resid norm > 1.933734977205e-03 ||r(i)||/||b|| 1.171935702734e-04 > 9 KSP preconditioned resid norm 7.267055499898e-02 true resid norm > 9.770926706361e-04 ||r(i)||/||b|| 5.921647997767e-05 > 10 KSP preconditioned resid norm 3.833602982459e-02 true resid norm > 9.695036607433e-04 ||r(i)||/||b|| 5.875654975214e-05 > velocity[0] took 10 iterations to reach residual 3.83360e-02 > Interesting that the true residual is dropping faster than preconditioned residual. Can you show the output from both runs? Are each individually reproducible? -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 4560 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Sun Jun 3 10:28:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 3 Jun 2012 10:28:27 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: <0C9CAFE8C4E5CA49884E636AE1D7CD690585859A@MAILBOX0B.psi.ch> References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD690585859A@MAILBOX0B.psi.ch> Message-ID: On Sun, Jun 3, 2012 at 10:20 AM, Niceno Bojan wrote: > I will need some time with -ksp_view_binary. > > The file being created (called "binaryoutput") seems to grow as simulation > time steps go by. It seems to me that all systems solved during a run are > stored in the same file. I am struggling to understand how to pick the one > I want to reproduce in ex10. > Yeah, you can either call MatView() and VecView() yourself or you can use PetscOptionsSetValue("-ksp_view_binary", "1") before the solve. You can also load the whole sequence in Matlab. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Jun 3 11:13:15 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 03 Jun 2012 18:13:15 +0200 Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in ksp In-Reply-To: References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com> <4FA3F056.4090406@gmail.com> <4FA4279F.1030204@gmail.com> <5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov> <4FA42CFC.8090002@gmail.com> <4FA44D5C.6010205@gmail.com> <4FA97132.3080103@gmail.com> <4FAD12FB.6080506@gmail.com> <4FAD195B.4030302@gmail.com> Message-ID: <4FCB8D1B.6060107@gmail.com> Hi, I have update the ex29f.F90 since there's some problems with MPI in the old version. Hopefully the DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90 can work in linux. Yours sincerely, TAY wee-beng On 11/5/2012 4:04 PM, John Mousel wrote: > Include both header files. > > #include > #include > > The h90 header only includes the interface definitions for Fortran. > > John > > On Fri, May 11, 2012 at 8:51 AM, TAY wee-beng > wrote: > > > > On 11/5/2012 3:30 PM, Matthew Knepley wrote: >> On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng > > wrote: >> >> Hi, >> >> I have been using the GUI environment to do debugging so I am >> a bit reluctant to learn Valgrind as its outputs seems a bit >> daunting. But I guess John is right. I've been spending >> these few days learning bit by bit. >> >> I realised that the error occurs in computerhs, at: >> >> >> I bet this is a beautiful Fortranism. Do you include the F90 >> header file with the interface definition? >> If not, Fortran just craps out like this. I can't stress enough >> how much time would be saved by >> switching languages to something with at least a modicum of error >> checking. > > I initially used: > > #include "finclude/petsc.h90" > > Compilation and linking was fine in Linux and vs2008. > > Now I changed it to what ex22.F was using : > > #include > #include > #include > #include > #include > #include > > Compiling was ok but linking failed in Linux and VS2008: > > undefined reference to `dmdavecgetarrayf90_' > > I tried changing #include to #include > and everything was ok in VS2008 again, > giving the right answers. > > However, in Linux, I got the following error: > > [wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c -fPIC > -g -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include > -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include > -I/opt/openmpi-1.5.3/include -o ex29f.o ex29f.F90 > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDAStencilType :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDAStencilType :: st > --------^ > ex29f.F90(68): error #6404: This name does not have a type, and > must have an explicit type. [DMDA_BOUNDARY_NONE] > call > DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > -----------------------------------^ > ex29f.F90(68): error #6404: This name does not have a type, and > must have an explicit type. [DMDA_STENCIL_STAR] > call > DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > -------------------------------------------------------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDAStencilType :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDAStencilType :: st > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDABoundaryType :: pt > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #5082: Syntax error, found '::' when expecting one of: ( % : > . = => > DMDAStencilType :: st > -------------------------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDABoundaryType :: pt > --------^ > /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): > error #6590: This statement is not permitted as a statement within > a derived-type-def > DMDAStencilType :: st > > Is there some errors in petscdmda.h90? > > > >> >> Matt >> >> *call DMDAVecRestoreArrayF90(da,b,array,ierr)* >> >> ==27464== Invalid write of size 8 >> ==27464== at 0x402835: computerhs_ (ex29f.F90:119) >> ==27464== Address 0xfffffffffffffef0 is not stack'd, >> malloc'd or (recently) free'd >> ==27464== >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >> Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC >> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> OS X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are >> not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of >> the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown >> [0]PETSC ERROR: [0] DMComputeFunction line 2085 >> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c >> [0]PETSC ERROR: [0] KSPSetUp line 182 >> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Signal received! >> >> I have checked that "array" 's values are correct. This >> statement executed without problems in VS2008. If I replace >> the above with something like: >> >> *call VecSet(b,Hy,ierr)* >> >> Everything is fine in Linux. >> >> Is there something wrong with *DMDAVecRestoreArrayF90*? >> >> My code in the area is: >> >> call >> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) >> >> call DMDAVecGetArrayF90(da,b,array,ierr) >> >> do j = ys,ys+ym-1 >> >> do i = xs,xs+xm-1 >> >> array(i,j) = >> exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy >> >> end do >> >> end do >> >> call DMDAVecRestoreArrayF90(da,b,array,ierr) >> >> call VecAssemblyBegin(b,ierr) >> >> call VecAssemblyEnd(b,ierr) >> >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> On 8/5/2012 9:41 PM, John Mousel wrote: >>> TAY wee-bing, >>> >>> If you want to be a programmer that writes interesting and >>> reliable code, you need to be willing to use the tools of >>> the trade. I can't think of a bigger time-saver than >>> Valgrind. I would suggest that you learn to use it and use >>> it a lot. I bet it will lead you to the root of your problem >>> pretty quickly. >>> >>> John >>> >>> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng >>> > wrote: >>> >>> Hi, >>> >>> I compiled and run my code under visual studio 2008 with >>> intel fortran. Everything works ok. >>> >>> However, when I tried to run the code in linux, I got >>> the error as below. The error happens when >>> KSPSetUp(ksp,ierr) is called. >>> >>> However, I am not able to print VecView or MatView to >>> view if there's any errors. Is there any recommendation >>> for debugging? I hope I do not need to valgrind if possible. >>> >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: >>> Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC >>> ERROR: or try http://valgrind.org on GNU/linux and Apple >>> Mac OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in >>> stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the >>> stack are not available, >>> [0]PETSC ERROR: INSTEAD the line number of the >>> start of the function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown >>> [0]PETSC ERROR: [0] DMComputeFunction line 2085 >>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c >>> [0]PETSC ERROR: [0] KSPSetUp line 182 >>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: --------------------- Error Message >>> ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: >>> 7ecdd63ec420b1659b960e65d96e822c5ac1a968 HG Date: Mon >>> May 07 21:42:26 2012 -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent >>> updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about >>> trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by >>> wtay Tue May 8 20:45:42 2012 >>> [0]PETSC ERROR: Libraries linked from >>> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib >>> [0]PETSC ERROR: Configure run at Tue May 8 10:47:59 2012 >>> [0]PETSC ERROR: Configure options >>> --with-mpi-dir=/opt/openmpi-1.5.3/ >>> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ >>> --with-debugging=1 --download-hypre=1 >>> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug >>> --known-mpi-shared=1 --with-shared-libraries >>> [0]PETSC ERROR: >>> ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in >>> unknown directory unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator >>> MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI >>> processes. >>> You may or may not see output from other processes, >>> depending on >>> exactly when Open MPI kills them. >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> On 5/5/2012 1:43 AM, Matthew Knepley wrote: >>>> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng >>>> > wrote: >>>> >>>> Hi, >>>> >>>> I wonder if you people are interested to include my >>>> ex29 fortran version in the petsc examples, which >>>> can help people who are using fortran. >>>> >>>> >>>> Yes, we will definitely include it. Please send the >>>> source, and a representative output with run options. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> Thanks. >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> On 4/5/2012 9:28 PM, Matthew Knepley wrote: >>>>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng >>>>> > wrote: >>>>> >>>>> >>>>> On 4/5/2012 9:16 PM, Barry Smith wrote: >>>>> >>>>> Do an hg pull and then run make in >>>>> src/mat/interface/ftn-custom/ >>>>> >>>>> Then it should link. >>>>> >>>>> Barry >>>>> >>>>> There was a E missing from the all caps >>>>> name of the function. >>>>> >>>>> After hg pull, I did: >>>>> >>>>> cd src//mat/interface/ftn-custom/ >>>>> >>>>> User at User-PC >>>>> /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom >>>>> $ make >>>>> make[1]: Warning: File >>>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)' >>>>> has modification time 787 s in the future >>>>> make[1]: Nothing to be done for `libc'. >>>>> make[1]: warning: Clock skew detected. Your >>>>> build may be incomplete. >>>>> >>>>> But it still can't work. >>>>> >>>>> >>>>> Something is messed up with the clock on this machine. >>>>> >>>>> HOWEVER, development requires certain basic skills >>>>> in order to debug your work. We >>>>> cannot be the ones debugging your code. Now >>>>> >>>>> nm $PETSC_ARCH/lib/libpetsc.a | grep -i >>>>> MatNullSpaceRemove >>>>> >>>>> will look for the symbol. >>>>> >>>>> Matt >>>>> >>>>> >>>>> >>>>> On May 4, 2012, at 2:11 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Fri, May 4, 2012 at 3:01 PM, TAY >>>>> wee-beng>>>> > wrote: >>>>> >>>>> On 4/5/2012 5:17 PM, Matthew Knepley >>>>> wrote: >>>>> >>>>> On Fri, May 4, 2012 at 11:05 AM, >>>>> TAY wee-beng>>>> > wrote: >>>>> >>>>> On 4/5/2012 3:05 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Fri, May 4, 2012 at 8:59 >>>>> AM, TAY >>>>> wee-beng>>>> > wrote: >>>>> >>>>> Hi, >>>>> >>>>> Is there anything else I can >>>>> try to get it working right? >>>>> >>>>> The MatGetNullSpaceRemove() is >>>>> missing. >>>>> >>>>> Where should I add >>>>> MatGetNullSpaceRemove and what are >>>>> its syntax? I googled but there's >>>>> no results. >>>>> >>>>> Fixed in p;etsc-dev: >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html >>>>> >>>>> I just compiled the updated petsc-dev >>>>> but I got the same error msg when I use: >>>>> >>>>> call >>>>> MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr) >>>>> >>>>> error LNK2019: unresolved external >>>>> symbol MATNULLSPACEREMOVE referenced >>>>> in function COMPUTERHS >>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : >>>>> fatal error LNK1120: 1 unresolved >>>>> externals >>>>> >>>>> That function is in: >>>>> >>>>> src/mat/interface/ftn-custom/zmatrixf.c >>>>> >>>>> Did that compile? Can you see the >>>>> symbol in libpetsc.a? >>>>> >>>>> Matt >>>>> >>>>> Matt >>>>> >>>>> Thanks. >>>>> >>>>> Matt >>>>> >>>>> Thanks? >>>>> >>>>> >>>>> On 2/5/2012 10:11 PM, Matthew >>>>> Knepley wrote: >>>>> >>>>> On Wed, May 2, 2012 at >>>>> 1:55 PM, TAY >>>>> wee-beng>>>> > >>>>> wrote: >>>>> Hi, >>>>> >>>>> I did a MatView and >>>>> VecView on both C and >>>>> Fortran, right after Mat >>>>> and Vec assembly. I have >>>>> attached the printout >>>>> below. They are exactly >>>>> the same, but yet the >>>>> result is different in >>>>> Neumann condition. >>>>> However, the dirichlet >>>>> condition gives the >>>>> correct ans. Is there >>>>> anything else that could >>>>> be wrong even if the Mat >>>>> and Vec are the same? >>>>> >>>>> Did you set the null space >>>>> for the matrix when you >>>>> have Neumann conditions? >>>>> >>>>> Yes, for the matrix, I set as: >>>>> >>>>> call >>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) >>>>> >>>>> call >>>>> MatSetNullSpace(jac,nullspace,ierr) >>>>> >>>>> call >>>>> MatNullSpaceDestroy(nullspace,ierr) >>>>> >>>>> for the Vec, >>>>> >>>>> call >>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) >>>>> >>>>> !call >>>>> MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr) >>>>> >>>>> call >>>>> MatNullSpaceDestroy(nullspace,ierr) >>>>> >>>>> MatNullSpaceRemove was comment >>>>> out because there's error >>>>> during linking >>>>> >>>>> >>>>> Matt >>>>> >>>>> Thanks! >>>>> >>>>> Fortran: >>>>> >>>>> Matrix Object: 1 MPI processes >>>>> type: seqaij >>>>> row 0: (0, 2) (1, -1) >>>>> (3, -1) >>>>> row 1: (0, -1) (1, 3) >>>>> (2, -1) (4, -1) >>>>> row 2: (1, -1) (2, 2) >>>>> (5, -1) >>>>> row 3: (0, -1) (3, 3) >>>>> (4, -1) (6, -1) >>>>> row 4: (1, -1) (3, -1) >>>>> (4, 4) (5, -1) (7, -1) >>>>> row 5: (2, -1) (4, -1) >>>>> (5, 3) (8, -1) >>>>> row 6: (3, -1) (6, 2) >>>>> (7, -1) >>>>> row 7: (4, -1) (6, -1) >>>>> (7, 3) (8, -1) >>>>> row 8: (5, -1) (7, -1) >>>>> (8, 2) >>>>> Vector >>>>> Object:Vec_0000000084000000_0 >>>>> 1 MPI processes >>>>> type: mpi >>>>> Process [0] >>>>> 0.25 >>>>> 0.0205213 >>>>> 1.135e-005 >>>>> 0.0205213 >>>>> 0.00168449 >>>>> 9.31663e-007 >>>>> 1.135e-005 >>>>> 9.31663e-007 >>>>> 5.15289e-010 >>>>> Vector >>>>> Object:Vec_0000000084000000_1 >>>>> 1 MPI processes >>>>> type: mpi >>>>> Process [0] >>>>> 0.14924 >>>>> 0.0242397 >>>>> -0.0260347 >>>>> 0.0242397 >>>>> -0.0256192 >>>>> -0.0400102 >>>>> -0.0260347 >>>>> -0.0400102 >>>>> -0.0400102 >>>>> Press any key to continue >>>>> . . . >>>>> >>>>> C: >>>>> >>>>> Matrix Object: 1 MPI processes >>>>> type: seqaij >>>>> row 0: (0, 2) (1, -1) >>>>> (3, -1) >>>>> row 1: (0, -1) (1, 3) >>>>> (2, -1) (4, -1) >>>>> row 2: (1, -1) (2, 2) >>>>> (5, -1) >>>>> row 3: (0, -1) (3, 3) >>>>> (4, -1) (6, -1) >>>>> row 4: (1, -1) (3, -1) >>>>> (4, 4) (5, -1) (7, -1) >>>>> row 5: (2, -1) (4, -1) >>>>> (5, 3) (8, -1) >>>>> row 6: (3, -1) (6, 2) >>>>> (7, -1) >>>>> row 7: (4, -1) (6, -1) >>>>> (7, 3) (8, -1) >>>>> row 8: (5, -1) (7, -1) >>>>> (8, 2) >>>>> Vector >>>>> Object:Vec_0x1d3b000_0 1 >>>>> MPI processes >>>>> type: mpi >>>>> Process [0] >>>>> 0.25 >>>>> 0.0205212 >>>>> 1.135e-05 >>>>> 0.0205212 >>>>> 0.00168449 >>>>> 9.31663e-07 >>>>> 1.135e-05 >>>>> 9.31663e-07 >>>>> 5.15288e-10 >>>>> Vector >>>>> Object:Vec_0x1d3b000_1 1 >>>>> MPI processes >>>>> type: mpi >>>>> Process [0] >>>>> 0.139311 >>>>> 0.0305751 >>>>> -0.0220633 >>>>> 0.0305751 >>>>> -0.0135158 >>>>> -0.042185 >>>>> -0.0220633 >>>>> -0.042185 >>>>> -0.058449 >>>>> >>>>> >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >>>>> On 1/5/2012 11:54 PM, >>>>> Matthew Knepley wrote: >>>>> >>>>> On Tue, May 1, 2012 at >>>>> 5:48 PM, TAY >>>>> wee-beng>>>> > >>>>> wrote: >>>>> Hi, >>>>> >>>>> Do you mean my method >>>>> is wrong? >>>>> >>>>> I am following the >>>>> template of ex22f, >>>>> >>>>> where the variables >>>>> are declared as : >>>>> >>>>> PetscScalar v(5) >>>>> >>>>> MatStencil >>>>> row(4),col(4,5) >>>>> >>>>> Hence, >>>>> >>>>> for the neumann BC >>>>> >>>>> num = 1 >>>>> >>>>> if >>>>> (j/=0) then >>>>> >>>>> >>>>> v(num) = -rho*HxdHy >>>>> >>>>> >>>>> col(MatStencil_i,num) = i >>>>> >>>>> >>>>> col(MatStencil_j,num) >>>>> = j-1 >>>>> >>>>> num >>>>> = num + 1 >>>>> >>>>> end if >>>>> >>>>> if >>>>> (i/=0) then >>>>> >>>>> >>>>> v(num) = -rho*HydHx >>>>> >>>>> >>>>> col(MatStencil_i,num) >>>>> = i-1 >>>>> >>>>> >>>>> col(MatStencil_j,num) = j >>>>> >>>>> num >>>>> = num + 1 >>>>> >>>>> end if >>>>> >>>>> if >>>>> (i/=mx-1) then >>>>> >>>>> >>>>> v(num) = -rho*HydHx >>>>> >>>>> >>>>> col(MatStencil_i,num) >>>>> = i+1 >>>>> >>>>> >>>>> col(MatStencil_j,num) = j >>>>> >>>>> num >>>>> = num + 1 >>>>> >>>>> end if >>>>> >>>>> if >>>>> (j/=my-1) then >>>>> >>>>> >>>>> v(num) = -rho*HxdHy >>>>> >>>>> >>>>> col(MatStencil_i,num) = i >>>>> >>>>> >>>>> col(MatStencil_j,num) >>>>> = j+1 >>>>> >>>>> num >>>>> = num + 1 >>>>> >>>>> end if >>>>> >>>>> v(num) >>>>> = >>>>> ((num-1)/2.0)*rho*(HxdHy >>>>> + HydHx) >>>>> >>>>> print *, v >>>>> >>>>> >>>>> col(MatStencil_i,num) = i >>>>> >>>>> >>>>> col(MatStencil_j,num) = j >>>>> >>>>> !num = >>>>> num + 1 >>>>> >>>>> call >>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr) >>>>> >>>>> I do not get any more >>>>> out of range error. >>>>> However,my ans is >>>>> still different from >>>>> that of ex29 in C. >>>>> >>>>> This is very simple. >>>>> You have an error in >>>>> your code. Checking it >>>>> is very simple: run >>>>> the code and >>>>> break in >>>>> MatSetValues(). Make >>>>> sure ex29 makes calls >>>>> with exactly the same >>>>> indices as your ex29f. >>>>> >>>>> Matt >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> -- >>>>> What most >>>>> experimenters take for >>>>> granted before they >>>>> begin their >>>>> experiments is >>>>> infinitely more >>>>> interesting than any >>>>> results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters >>>>> take for granted before >>>>> they begin their >>>>> experiments is infinitely >>>>> more interesting than any >>>>> results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take >>>>> for granted before they begin >>>>> their experiments is >>>>> infinitely more interesting >>>>> than any results to which >>>>> their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for >>>>> granted before they begin their >>>>> experiments is infinitely more >>>>> interesting than any results to >>>>> which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for >>>>> granted before they begin their >>>>> experiments is infinitely more >>>>> interesting than any results to which >>>>> their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before >>>>> they begin their experiments is infinitely more >>>>> interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ! Concepts: KSP^solving a system of linear equations ! Concepts: KSP^Laplacian, 2d ! Processors: n !Added by Wee Beng Tay (W.B.Tay at tudelft.nl). !conversion from ex29 C version to Fortran !Inhomogeneous Laplacian in 2D. Modeled by the partial differential equation ! -div \rho grad u = f, 0 < x,y < 1, !with forcing function ! f = e^{-x^2/\nu} e^{-y^2/\nu} !with Dirichlet boundary conditions ! u = f(x,y) for x = 0, x = 1, y = 0, y = 1 !or pure Neumman boundary conditions !This uses multigrid to solve the linear system program ex29f implicit none ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! Include files ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ! ! petscsys.h - base PETSc routines petscvec.h - vectors ! petscmat.h - matrices ! petscksp.h - Krylov subspace methods petscpc.h - preconditioners !#include !#include !#include !#include !#include !#include !#include #include "finclude/petsc.h90" PetscErrorCode ierr DM da KSP ksp Vec x external ComputeRHS,ComputeMatrix PetscInt i1,i3,bc_cond !>bc_cond = boundary condition = 1/2 = dirichlet/neumann !>have to change bc_cond at 3 locations call PetscInitialize(PETSC_NULL_CHARACTER,ierr) bc_cond = 1 i3 = -3 i1 = 1 call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) call DMSetFunction(da,ComputeRHS,ierr) call DMSetJacobian(da,ComputeMatrix,ierr) call KSPSetDM(ksp,da,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSetUp(ksp,ierr) call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) call KSPGetSolution(ksp,x,ierr) call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) call KSPDestroy(ksp,ierr) call DMDestroy(da,ierr) call PetscFinalize(ierr) end program ex29f subroutine ComputeRHS(da,x,b,ierr) implicit none !#include !#include !#include !#include !#include !#include !#include #include "finclude/petsc.h90" PetscErrorCode ierr PetscInt i,j,ij,mx,my,xm,ym,xs,ys,bc_cond,status,myid,ij_sta,ij_end PetscScalar h,nu,rho PetscScalar Hx,Hy PetscScalar,pointer :: array(:,:) !Problem with using DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90 below in Vec x,b !linux with ifort, so VecSetValues was used. Worked in VS2008 though DM da MatNullSpace nullspace !>neumann BC integer, allocatable:: ij_array(:) real(8), allocatable :: arrays(:) nu = 0.1 bc_cond = 1 call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) Hx = 1.d0 / (mx-1) Hy = 1.d0 / (my-1) call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) call VecGetOwnershipRange(b,ij_sta,ij_end,ierr) allocate (ij_array(ij_sta:ij_end-1), STAT=status) if (status/=0) STOP "Cannot allocate memory 1" allocate (arrays(ij_sta:ij_end-1), STAT=status) if (status/=0) STOP "Cannot allocate memory 2" !Section below not usable in linux ifort yet - only works in VS2008 + ifort !call DMDAVecGetArrayF90(da,b,array,ierr) !do j = ys,ys+ym-1 ! do i = xs,xs+xm-1 ! array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy ! end do !end do !call DMDAVecRestoreArrayF90(da,b,array,ierr) !Section above not usable in linux ifort yet - only works in VS2008 + ifort ij = ij_sta - 1 do j = ys,ys+ym-1 do i = xs,xs+xm-1 ij = ij + 1 ij_array(ij) = ij arrays(ij) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy end do end do call VecSetValues(b,ij_end-ij_sta,ij_array,arrays,INSERT_VALUES,ierr) call VecAssemblyBegin(b,ierr) call VecAssemblyEnd(b,ierr) !call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr) if (bc_cond == 2) then call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) call MatNullSpaceRemove(nullspace,b,PETSC_NULL_OBJECT,ierr) call MatNullSpaceDestroy(nullspace,ierr) end if end subroutine ComputeRHS subroutine ComputeMatrix(da,x,JJ,jac,str,ierr) implicit none !#include !#include !#include !#include !#include #include "finclude/petsc.h90" Mat jac,JJ PetscErrorCode ierr DM da PetscInt i,j,k,mx,my,xm,num PetscInt ym,xs,ys,i1,i5,bc_cond PetscScalar v(5),Hx,Hy,rho,centerRho PetscScalar HydHx PetscScalar HxdHy MatStencil row(4),col(4,5) Vec x MatStructure str MatNullSpace nullspace !>neumann BC bc_cond = 1 rho = 1.0 i1 = 1 i5 = 5 centerRho = rho call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr) Hx = 1.d0 / (mx-1) Hy = 1.d0 / (my-1) HxdHy = Hx/Hy HydHx = Hy/Hx call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr) do j=ys,ys+ym-1 do i=xs,xs+xm-1 row(MatStencil_i) = i row(MatStencil_j) = j call ComputeRho(i,j,mx,my,centerRho,rho) if (i==0 .or. j==0 .or. i==mx-1 .or. j==my-1) then if (bc_cond == 1) then v(1) = 2.0*rho*(HxdHy + HydHx) call MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr) else if (bc_cond == 2) then num = 1 if (j/=0) then v(num) = -rho*HxdHy col(MatStencil_i,num) = i col(MatStencil_j,num) = j-1 num = num + 1 end if if (i/=0) then v(num) = -rho*HydHx col(MatStencil_i,num) = i-1 col(MatStencil_j,num) = j num = num + 1 end if if (i/=mx-1) then v(num) = -rho*HydHx col(MatStencil_i,num) = i+1 col(MatStencil_j,num) = j num = num + 1 end if if (j/=my-1) then v(num) = -rho*HxdHy col(MatStencil_i,num) = i col(MatStencil_j,num) = j+1 num = num + 1 end if v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx) col(MatStencil_i,num) = i col(MatStencil_j,num) = j !num = num + 1 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr) end if else v(1) = -rho*HxdHy col(MatStencil_i,1) = i col(MatStencil_j,1) = j-1 v(2) = -rho*HydHx col(MatStencil_i,2) = i-1 col(MatStencil_j,2) = j v(3) = 2.0*rho*(HxdHy + HydHx) col(MatStencil_i,3) = i col(MatStencil_j,3) = j v(4) = -rho*HydHx col(MatStencil_i,4) = i+1 col(MatStencil_j,4) = j v(5) = -rho*HxdHy col(MatStencil_i,5) = i col(MatStencil_j,5) = j+1 call MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr) end if end do end do call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr) !call MatView(jac,PETSC_VIEWER_STDOUT_WORLD,ierr) if (bc_cond == 2) then call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr) call MatSetNullSpace(jac,nullspace,ierr) call MatNullSpaceDestroy(nullspace,ierr) end if end subroutine ComputeMatrix subroutine ComputeRho(i,j,mx,my,centerRho,rho) PetscInt i,j,mx,my PetscScalar rho,centerRho if ((i > mx/3.0) .and. (i < 2.0*mx/3.0) .and. (j > my/3.0) .and. (j < 2.0*my/3.0)) then rho = centerRho else rho = 1.0 end if end subroutine ComputeRho From amesga1 at tigers.lsu.edu Sun Jun 3 11:20:31 2012 From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad) Date: Sun, 3 Jun 2012 11:20:31 -0500 Subject: [petsc-users] Solutions different after program restart In-Reply-To: References: <4FCB35CF.6040505@psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858596@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858598@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD6905858599@MAILBOX0B.psi.ch> <0C9CAFE8C4E5CA49884E636AE1D7CD690585859A@MAILBOX0B.psi.ch> Message-ID: <8CEF4DC1-7726-4236-B391-9D102FBD63F5@tigers.lsu.edu> Dear Bojan, I'm not sure if this relates to your problem but I had a similar situation: I use KSPSetInitialGuessNonzero with GMRES. Setting DIFFERENT_NONZERO_PATTERN in KSPSetOperators solved my problem. Best, Ata; On Jun 3, 2012, at 10:28 AM, Jed Brown wrote: > On Sun, Jun 3, 2012 at 10:20 AM, Niceno Bojan wrote: > I will need some time with -ksp_view_binary. > > The file being created (called "binaryoutput") seems to grow as simulation time steps go by. It seems to me that all systems solved during a run are stored in the same file. I am struggling to understand how to pick the one I want to reproduce in ex10. > > Yeah, you can either call MatView() and VecView() yourself or you can use PetscOptionsSetValue("-ksp_view_binary", "1") before the solve. You can also load the whole sequence in Matlab. -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Jun 4 00:03:13 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 4 Jun 2012 08:33:13 +0330 Subject: [petsc-users] about Matrix-Free Message-ID: Dear all, I want to compute Jacobian matrix via Matrix-Free manner in SNES procedure. I should consider the time-marching term as M*dQ/dt with is not the part of finite-differencing and should be added into diagonal blocks of the matrix. So, how should be right to calculate the Jacobian using the prepared Petsc routines? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Jun 4 01:41:06 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 4 Jun 2012 10:11:06 +0330 Subject: [petsc-users] about Matrix-Free In-Reply-To: References: Message-ID: According to my last question, I am trying to use Shell to handle the MatrixFree procedure as below: 1- Define residual as: F := M * DQ/Dt - R(Q) (R is the nonlinear residual obtained from discretization) 2- MatCreateShell 3- MatShellSetOperation -> MATOP_MULT as follows: dF/dQ * v = M * v / Dt - ( R(Q+eps*v) - R(Q) ) / eps 4- Using this operation in each step of KSP (GMRES) subspace construction. Does the above procedure seem reasonable to use with SNES? Please help me if it needs more considerations. Thanks, BehZad On Mon, Jun 4, 2012 at 8:33 AM, behzad baghapour wrote: > Dear all, > > I want to compute Jacobian matrix via Matrix-Free manner in SNES > procedure. I should consider the time-marching term as M*dQ/dt with is not > the part of finite-differencing and should be added into diagonal blocks of > the matrix. So, how should be right to calculate the Jacobian using the > prepared Petsc routines? > > Thanks a lot, > BehZad > -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Jun 4 02:36:41 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 4 Jun 2012 07:36:41 +0000 Subject: [petsc-users] GMRES pc right fails while pc left succeeds Message-ID: > > To solve the incompressible Navier-Stokes equations, I'm using > > cell-centered finite volumes and Picard linearization. At each > > non-linear iteration, I'm using one linear (Krylov) iteration with a > > SIMPLE-type block preconditioner (for example Elman e.a, JCP 227, > > 2008). The matrix and preconditioner are shells. Below, I'm showing > > the results for the first few non-linear iterations using > > > > (1) KSPRICHARDSON to check that the shells are ok > > (2) GMRES with right preconditioning > > (3) GMRES with left preconditioning > > > > To my surprise (2) fails while (1) and (3) succeed. The reason I'm > > interested in right preconditioning is that SIMPLE is a variable > > preconditioner so in theory I should be using FGMRES as soon as I > > increase the number of linear (Krylov) iterations. > > > > One inner iteration is already nonlinear. > > What is the inner iteration here? SIMPLE uses a diagonal approximation of > the (velocity,velocity) block in an explicit approximation of the Schur > complement (which you could form using the matrix product). Yes, the diagonal of the velocity block is used to form the Schur complement approximation explicitly. I'm solving the velocity subsystem and the Schur complement subsystem to a relative tolerance of 0.01. Decreasing this tolerance does not change the behaviour of GMRES with right preconditioning. > > Is there a reason you aren't using PCFIELDSPLIT for this? > Historical reasons. If I had to redo the whole thing I would probably use PCFIELDSPLIT. > > > What could be the > > reason that GMRES fails with right pc? > > > > Both might be failing, you'll have to show that a tight tolerance can > eventually be reached. What does FGMRES do? Well, I only showed the first few non-linear iterations, but eventually RICHARDSON and GMRES with left pc go from 1e+7 to 1e-3 in the preconditioned norm and from 1e+2 to 1e-5 in the true norm. FGMRES behaves like GMRES with right pc, totally stuck. > > Note that the preconditioned norm (see Richardson and left-GMRES) becomes > huge. A preconditioned residual that is 10^6 larger than unpreconditioned > usually means the preconditioner is incorrect. Yes, it's remarkably large in the beginning. At convergence the difference is about 10^2. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From w_ang_temp at 163.com Mon Jun 4 02:58:31 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 4 Jun 2012 15:58:31 +0800 (CST) Subject: [petsc-users] About the ./configure of petsc and mpi In-Reply-To: References: <3629d52d.1b524.137ad137a1a.Coremail.w_ang_temp@163.com> <7233155c.1f05d.137ad89324c.Coremail.w_ang_temp@163.com> <36a9551f.1c59b.137ad9909f1.Coremail.w_ang_temp@163.com> Message-ID: <1513b663.7053.137b6809e39.Coremail.w_ang_temp@163.com> Hello, Jed The problem has been solved. Just as you said, the variable 'LTR ' shoud be initialized. Thanks. Jim >? 2012-06-02 22:34:58?"Jed Brown" ??? >On Sat, Jun 2, 2012 at 9:28 AM, w_ang_temp wrote: >Hello Jed > It is true that LTR is uninitialized. >There are lots of ways for uninitialized variables to accidentally have valid values. That doesn't mean your program is correct, it just means that it >accidentally "worked". You have to fix your code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 4 06:34:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 06:34:03 -0500 Subject: [petsc-users] about Matrix-Free In-Reply-To: References: Message-ID: On Mon, Jun 4, 2012 at 1:41 AM, behzad baghapour wrote: > According to my last question, I am trying to use Shell to handle the > MatrixFree procedure as below: > > 1- Define residual as: > F := M * DQ/Dt - R(Q) (R is the nonlinear residual obtained from > discretization) > 2- MatCreateShell > 3- MatShellSetOperation -> MATOP_MULT as follows: > dF/dQ * v = M * v / Dt - ( R(Q+eps*v) - R(Q) ) / eps > 4- Using this operation in each step of KSP (GMRES) subspace construction. > > Does the above procedure seem reasonable to use with SNES? Please help me > if it needs more considerations. > You can just set the SNES function and do -snes_mf, then this will be done for you. You might also consider using TS. TSSetIFunction() is the interface you propose. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 4 06:39:20 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 06:39:20 -0500 Subject: [petsc-users] GMRES pc right fails while pc left succeeds In-Reply-To: References: Message-ID: On Mon, Jun 4, 2012 at 2:36 AM, Klaij, Christiaan wrote: > Yes, the diagonal of the velocity block is used to form the Schur > complement approximation explicitly. I'm solving the velocity > subsystem and the Schur complement subsystem to a relative tolerance > of 0.01. Decreasing this tolerance does not change the behaviour of > GMRES with right preconditioning. > Just to be sure, are both of these approximations in the preconditioner or have you absorbed part into the "operator"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Mon Jun 4 07:37:31 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 4 Jun 2012 20:37:31 +0800 (CST) Subject: [petsc-users] A qustion of PETSc In-Reply-To: <2d3e7f56.223bf.13789ae3c60.Coremail.w_ang_temp@163.com> References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com> <5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com> <74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com> <2d3e7f56.223bf.13789ae3c60.Coremail.w_ang_temp@163.com> Message-ID: <717f3e5a.11ea2.137b7800db3.Coremail.w_ang_temp@163.com> Hello As I become familiar with PETSc, I make a conclusion about the question and hope that it is right. In order to get satisfactory results when solving a linear system with KSP, the Krylov space methods are typically used in conjunction with a preconditioner. So the user should select the effective Krylov method(such as cg) and preconditioner(such as jacobi) for this linear system Ax=b. Or it maybe not convergence. Thanks. Jim >At 2012-05-26 23:05:26,w_ang_temp wrote: >Hello, Matt > The results of the parallel and the serial are identical by the "-ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual". > I will make a carefulanalysis of these parameters next day because it is late night here now. > Thanks again. Jim >? 2012-05-26 22:26:20?"Matthew Knepley" ??? >On Sat, May 26, 2012 at 2:22 PM, w_ang_temp wrote: >Hello,Matt > Thanks again. > What you said("Use -ksp_type preonly -pc_type lu") means that I should use a direct solver.And I am puzzled. >Okay, you want to remove variables from the investigation. You say there is a difference between >the parallel and serial solve. We do this all the time, so it is not a bug. It is an issue of understanding. >First, take away the tolerance issues: > -ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual >and send output of the serial and parallel run. > Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 4 07:49:21 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 07:49:21 -0500 Subject: [petsc-users] A qustion of PETSc In-Reply-To: <717f3e5a.11ea2.137b7800db3.Coremail.w_ang_temp@163.com> References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com> <5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com> <74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com> <2d3e7f56.223bf.13789ae3c60.Coremail.w_ang_temp@163.com> <717f3e5a.11ea2.137b7800db3.Coremail.w_ang_temp@163.com> Message-ID: Yes, a good linear solver involves a suitable choice of Krylov method and preconditioner. On Mon, Jun 4, 2012 at 7:37 AM, w_ang_temp wrote: > Hello > As I become familiar with PETSc, I make a conclusion about the > question and hope that it is right. > In order to get satisfactory results when solving a linear system > with KSP, the Krylov space methods > are typically used in conjunction with a preconditioner. So the user > should select the effective Krylov > method(such as cg) and preconditioner(such as jacobi) for this linear > system Ax=b. Or it maybe not > convergence. > Thanks. > Jim > > > >At 2012-05-26 23:05:26,w_ang_temp wrote: > > >Hello, Matt > > The results of the parallel and the serial are identical by the > "-ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual". > > I will make a careful analysis of these parameters next day > because it is late night here now. > > Thanks again. > Jim > >? 2012-05-26 22:26:20?"Matthew Knepley" ??? > > >On Sat, May 26, 2012 at 2:22 PM, w_ang_temp wrote: > >> >Hello,Matt >> > Thanks again. >> > What you said("Use -ksp_type preonly -pc_type lu") means that I >> should use a direct solver.And I am puzzled. >> > > >Okay, you want to remove variables from the investigation. You say there > is a difference between > >the parallel and serial solve. We do this all the time, so it is not a > bug. It is an issue of understanding. > >First, take away the tolerance issues: > > > -ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual > > >and send output of the serial and parallel run. > > > Matt > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Mon Jun 4 08:36:39 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 4 Jun 2012 17:06:39 +0330 Subject: [petsc-users] about Matrix-Free In-Reply-To: References: Message-ID: OK. Thank you. On Mon, Jun 4, 2012 at 3:04 PM, Jed Brown wrote: > On Mon, Jun 4, 2012 at 1:41 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> According to my last question, I am trying to use Shell to handle the >> MatrixFree procedure as below: >> >> 1- Define residual as: >> F := M * DQ/Dt - R(Q) (R is the nonlinear residual obtained from >> discretization) >> 2- MatCreateShell >> 3- MatShellSetOperation -> MATOP_MULT as follows: >> dF/dQ * v = M * v / Dt - ( R(Q+eps*v) - R(Q) ) / eps >> 4- Using this operation in each step of KSP (GMRES) subspace >> construction. >> >> Does the above procedure seem reasonable to use with SNES? Please help me >> if it needs more considerations. >> > > You can just set the SNES function and do -snes_mf, then this will be done > for you. > > You might also consider using TS. TSSetIFunction() is the interface you > propose. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Jun 4 08:45:07 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 4 Jun 2012 13:45:07 +0000 Subject: [petsc-users] GMRES pc right fails while pc left succeeds Message-ID: > > Yes, the diagonal of the velocity block is used to form the Schur > > complement approximation explicitly. I'm solving the velocity > > subsystem and the Schur complement subsystem to a relative tolerance > > of 0.01. Decreasing this tolerance does not change the behaviour of > > GMRES with right preconditioning. > > > > Just to be sure, are both of these approximations in the preconditioner or > have you absorbed part into the "operator"? Both approximations are in the preconditioner. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From jedbrown at mcs.anl.gov Mon Jun 4 08:48:34 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 08:48:34 -0500 Subject: [petsc-users] GMRES pc right fails while pc left succeeds In-Reply-To: References: Message-ID: On Mon, Jun 4, 2012 at 8:45 AM, Klaij, Christiaan wrote: > Both approximations are in the preconditioner. I'm afraid I don't have an explanation based on the information you have sent. If you can set up a test case that shows the issue, I'll investigate further. I would think about null space issues. For example, if the Schur complement has constants in its null space, but the right hand side is not consistent and/or the KSP was not informed of the null space. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jiangwen84 at gmail.com Mon Jun 4 09:02:06 2012 From: jiangwen84 at gmail.com (Wen Jiang) Date: Mon, 4 Jun 2012 10:02:06 -0400 Subject: [petsc-users] petsc dev mat assembly fails In-Reply-To: References: Message-ID: Hi, Sorry for the confusion. >>There aren't any errors in this -info output? What did you see? Send the >>whole error message. The error message is attached. It said Argument out of range and New nonzero at () caused a malloc. I did not have such error in petsc 3.2. I realized that I did not give exact preallocation in MatCreateAIJ, but I did not set its preallocation call MatSetOption(mat, NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE). Does petsc dev automatically call MatSetOption? Regards, Wen -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vesicle.err Type: application/octet-stream Size: 34680 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Mon Jun 4 09:11:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 09:11:58 -0500 Subject: [petsc-users] petsc dev mat assembly fails In-Reply-To: References: Message-ID: On Mon, Jun 4, 2012 at 9:02 AM, Wen Jiang wrote: > Hi, > > Sorry for the confusion. > > >>There aren't any errors in this -info output? What did you see? Send the > >>whole error message. > > The error message is attached. It said Argument out of range and New > nonzero at () caused a malloc. I did not have such error in petsc 3.2. I > realized that I did not give exact preallocation in MatCreateAIJ, but I did > not set its preallocation call MatSetOption(mat, > NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE). Does petsc dev automatically call > MatSetOption? > "Preallocation routines now automatically set MAT_NEW_NONZERO_ALLOCATION_ERR, if you intentionally preallocate less than necessary then use MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) to disable the error generation." http://www.mcs.anl.gov/petsc/documentation/changes/dev.html In the future, ALWAYS send the full error message so we don't have to ask for it every time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Mon Jun 4 09:47:28 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Mon, 4 Jun 2012 14:47:28 +0000 Subject: [petsc-users] GMRES pc right fails while pc left succeeds Message-ID: > > Both approximations are in the preconditioner. > > > I'm afraid I don't have an explanation based on the information you have > sent. If you can set up a test case that shows the issue, I'll investigate > further. Thanks for offering, unfortunately I don't have a small testcase that shows the same behaviour. It only happens in the odd industrial case, convergence is fine for most similar cases and perfect for small academic cases. > > I would think about null space issues. For example, if the Schur complement > has constants in its null space, but the right hand side is not consistent > and/or the KSP was not informed of the null space. Definitely not the constant, but there could be a near null space... dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From zonexo at gmail.com Mon Jun 4 16:11:37 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 04 Jun 2012 23:11:37 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid Message-ID: <4FCD2489.5000909@gmail.com> Hi, I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. ex29 to Fortran and then added into my code to solve the Poisson equation. The main subroutines are: call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) call DMSetFunction(da,ComputeRHS,ierr) call DMSetJacobian(da,ComputeMatrix,ierr) call KSPSetDM(ksp,da,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSetUp(ksp,ierr) call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) call KSPGetSolution(ksp,x,ierr) call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) call KSPDestroy(ksp,ierr) call DMDestroy(da,ierr) call PetscFinalize(ierr) Since the LHS matrix doesn't change, I only set up at the 1st time step, thereafter I only called ComputeRHS every time step. I was using HYPRE's geometric multigrid and the speed was much faster. What other options can I tweak to improve the speed? Or should I call the subroutines above at every timestep? Thanks! -- Yours sincerely, TAY wee-beng From jedbrown at mcs.anl.gov Mon Jun 4 16:15:15 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 4 Jun 2012 16:15:15 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FCD2489.5000909@gmail.com> References: <4FCD2489.5000909@gmail.com> Message-ID: Always send -log_summary when asking about performance. On Mon, Jun 4, 2012 at 4:11 PM, TAY wee-beng wrote: > Hi, > > I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. > ex29 to Fortran and then added into my code to solve the Poisson equation. > > The main subroutines are: > > call KSPCreate(PETSC_COMM_WORLD,**ksp,ierr) > > call DMDACreate2d(PETSC_COMM_WORLD,**DMDA_BOUNDARY_NONE,DMDA_** > BOUNDARY_NONE,DMDA_STENCIL_**STAR,i3,i3,PETSC_DECIDE,PETSC_** > DECIDE,i1,i1,PETSC_NULL_**INTEGER,PETSC_NULL_INTEGER,da,**ierr) > call DMSetFunction(da,ComputeRHS,**ierr) > call DMSetJacobian(da,**ComputeMatrix,ierr) > call KSPSetDM(ksp,da,ierr) > > call KSPSetFromOptions(ksp,ierr) > call KSPSetUp(ksp,ierr) > call KSPSolve(ksp,PETSC_NULL_**OBJECT,PETSC_NULL_OBJECT,ierr) > call KSPGetSolution(ksp,x,ierr) > call VecView(x,PETSC_VIEWER_STDOUT_**WORLD,ierr) > call KSPDestroy(ksp,ierr) > call DMDestroy(da,ierr) > call PetscFinalize(ierr) > > > Since the LHS matrix doesn't change, I only set up at the 1st time step, > thereafter I only called ComputeRHS every time step. > > I was using HYPRE's geometric multigrid and the speed was much faster. > > What other options can I tweak to improve the speed? Or should I call the > subroutines above at every timestep? > > Thanks! > > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 4 18:34:22 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 4 Jun 2012 18:34:22 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: References: <4FCD2489.5000909@gmail.com> Message-ID: <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> Also run with -ksp_view to see exasctly what solver options it is using. For example the number of levels, smoother on each level etc. My guess is that the below is running on one level (because I don't see you supplying options to control the number of levels etc). Barry On Jun 4, 2012, at 4:15 PM, Jed Brown wrote: > Always send -log_summary when asking about performance. > > On Mon, Jun 4, 2012 at 4:11 PM, TAY wee-beng wrote: > Hi, > > I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. ex29 to Fortran and then added into my code to solve the Poisson equation. > > The main subroutines are: > > call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) > > call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > call DMSetFunction(da,ComputeRHS,ierr) > call DMSetJacobian(da,ComputeMatrix,ierr) > call KSPSetDM(ksp,da,ierr) > > call KSPSetFromOptions(ksp,ierr) > call KSPSetUp(ksp,ierr) > call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) > call KSPGetSolution(ksp,x,ierr) > call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) > call KSPDestroy(ksp,ierr) > call DMDestroy(da,ierr) > call PetscFinalize(ierr) > > > Since the LHS matrix doesn't change, I only set up at the 1st time step, thereafter I only called ComputeRHS every time step. > > I was using HYPRE's geometric multigrid and the speed was much faster. > > What other options can I tweak to improve the speed? Or should I call the subroutines above at every timestep? > > Thanks! > > > -- > Yours sincerely, > > TAY wee-beng > > From bsmith at mcs.anl.gov Mon Jun 4 18:39:24 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 4 Jun 2012 18:39:24 -0500 Subject: [petsc-users] GMRES pc right fails while pc left succeeds In-Reply-To: References: Message-ID: In theory if you use petsc-3.3 (release any day) switching to use the PCFIELDSPLIT solver (with appropriate options) should be able to reproduce your algorithm (I think) with very little coding on your part. This would make it easier for us to help you understand the issue since we know exactly what PCFIELDSPLIT does and how it is suppose to work and the -ksp_view option (or -snes_view) tells exactly the algorithm being used. Barry On Jun 4, 2012, at 9:47 AM, Klaij, Christiaan wrote: >>> Both approximations are in the preconditioner. >> >> >> I'm afraid I don't have an explanation based on the information you have >> sent. If you can set up a test case that shows the issue, I'll investigate >> further. > > Thanks for offering, unfortunately I don't have a small testcase that > shows the same behaviour. It only happens in the odd industrial case, > convergence is fine for most similar cases and perfect for small > academic cases. > >> >> I would think about null space issues. For example, if the Schur complement >> has constants in its null space, but the right hand side is not consistent >> and/or the KSP was not informed of the null space. > > Definitely not the constant, but there could be a near null space... > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > From thomas.witkowski at tu-dresden.de Tue Jun 5 01:48:54 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 05 Jun 2012 08:48:54 +0200 Subject: [petsc-users] Set vector elements after VecAssembly? Message-ID: <4FCDABD6.3060209@tu-dresden.de> Is it possible to update some vector entries after VecAssembly has been called? Is due to some boundary conditions. In principle, I need something that is also done in MatZeroRows on the last two vector parameters, but without the need to provide some matrix information. Is this possible? Thomas From samuelandjw at gmail.com Tue Jun 5 04:59:04 2012 From: samuelandjw at gmail.com (Degang Wu) Date: Tue, 05 Jun 2012 17:59:04 +0800 Subject: [petsc-users] use mpic++ instead of mpicc Message-ID: <4FCDD868.7060807@gmail.com> Hi, I am new to Petsc and Slepc, and I rely on the makefile provided in the example code folders to compile. For example, the makefile in src/eps/examples/tutorials/makefile. How should I modify the makefile so that make will use mpic++ rather than mpicc to compile and link? Wu Degang From jedbrown at mcs.anl.gov Tue Jun 5 06:56:50 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 5 Jun 2012 06:56:50 -0500 Subject: [petsc-users] Set vector elements after VecAssembly? In-Reply-To: <4FCDABD6.3060209@tu-dresden.de> References: <4FCDABD6.3060209@tu-dresden.de> Message-ID: On Tue, Jun 5, 2012 at 1:48 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Is it possible to update some vector entries after VecAssembly has been > called? If you use VecSetValues(), call VecAssemblyBegin/End again when you are done. You can also use VecGetArray() (or variants) to manipulate vector entries without needing to call VecAssemblyBegin/End. > Is due to some boundary conditions. In principle, I need something that is > also done in MatZeroRows on the last two vector parameters, but without the > need to provide some matrix information. Is this possible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Tue Jun 5 08:29:03 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 5 Jun 2012 15:29:03 +0200 Subject: [petsc-users] use mpic++ instead of mpicc In-Reply-To: <4FCDD868.7060807@gmail.com> References: <4FCDD868.7060807@gmail.com> Message-ID: <37981759-473F-4EFF-9270-DE79F5383E97@dsic.upv.es> El 05/06/2012, a las 11:59, Degang Wu escribi?: > Hi, > > I am new to Petsc and Slepc, and I rely on the makefile provided in the example code folders to compile. For example, the makefile in src/eps/examples/tutorials/makefile. How should I modify the makefile so that make will use mpic++ rather than mpicc to compile and link? > > Wu Degang You must configure PETSc --with-clanguage=c++ Jose From renzhengyong at gmail.com Tue Jun 5 12:48:41 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Tue, 5 Jun 2012 19:48:41 +0200 Subject: [petsc-users] about the preconditioned GMRES Message-ID: Dear Petsc users, My purpose is to solve AmatX=b, A is a symmetric indefinite matrix. Here, I read the manual about the usage of preconditioned GMRES method: Given a matrix Pmat which is a good approximation to Amat, and call PetscErrorCode KSPSetOperators(KSP ksp,Mat Amat,Mat Pmat,MatStructure flag). My question is: If I have an explicit good approximation (T) to the inverse of the Amat (that is T=Pmat^{_1}), which call will be used to tell the KSP of having this explicit preconditioner. Thanks in advance for your suggestions. Best wishes Zhengyong -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Jun 5 12:56:51 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 5 Jun 2012 12:56:51 -0500 Subject: [petsc-users] about the preconditioned GMRES In-Reply-To: References: Message-ID: ZhengYong : You may use shell preconditioner for your own PC. See ~petsc/src/ksp/ksp/examples/tutorials/ex15.c Hong Dear Petsc users, > > My purpose is to solve AmatX=b, A is a symmetric indefinite matrix. > Here, I read the manual about the usage of preconditioned GMRES method: > Given a matrix Pmat which is a good approximation to Amat, and call > > PetscErrorCode KSPSetOperators(KSP ksp,Mat Amat,Mat Pmat,MatStructure flag). > > My question is: If I have an explicit good approximation (T) to the inverse of the Amat (that is T=Pmat^{_1}), > which call will be used to tell the KSP of having this explicit preconditioner. > > Thanks in advance for your suggestions. > > Best wishes > Zhengyong > > > > -- > Zhengyong Ren > AUG Group, Institute of Geophysics > Department of Geosciences, ETH Zurich > NO H 47 Sonneggstrasse 5 > CH-8092, Z?rich, Switzerland > Tel: +41 44 633 37561 > e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > Gmail: renzhengyong at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 5 13:01:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 5 Jun 2012 13:01:57 -0500 Subject: [petsc-users] about the preconditioned GMRES In-Reply-To: References: Message-ID: On Tue, Jun 5, 2012 at 12:48 PM, RenZhengYong wrote: > Dear Petsc users, > > My purpose is to solve AmatX=b, A is a symmetric indefinite matrix. > Here, I read the manual about the usage of preconditioned GMRES method: > Given a matrix Pmat which is a good approximation to Amat, and call > > PetscErrorCode KSPSetOperators(KSP ksp,Mat Amat,Mat Pmat,MatStructure flag). > > My question is: If I have an explicit good approximation (T) to the inverse of the Amat (that is T=Pmat^{_1}), > which call will be used to tell the KSP of having this explicit preconditioner. > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMAT.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Tue Jun 5 13:21:06 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Tue, 5 Jun 2012 20:21:06 +0200 Subject: [petsc-users] about the preconditioned GMRES In-Reply-To: References: Message-ID: Hi, Hong and Jed, Thanks a lot as usual. Best wishes Zhengyong On Tue, Jun 5, 2012 at 8:01 PM, Jed Brown wrote: > On Tue, Jun 5, 2012 at 12:48 PM, RenZhengYong wrote: > >> Dear Petsc users, >> >> My purpose is to solve AmatX=b, A is a symmetric indefinite matrix. >> Here, I read the manual about the usage of preconditioned GMRES method: >> Given a matrix Pmat which is a good approximation to Amat, and call >> >> PetscErrorCode KSPSetOperators(KSP ksp,Mat Amat,Mat Pmat,MatStructure flag). >> >> >> My question is: If I have an explicit good approximation (T) to the inverse of the Amat (that is T=Pmat^{_1}), >> which call will be used to tell the KSP of having this explicit preconditioner. >> >> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCMAT.html > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Tue Jun 5 14:02:02 2012 From: john.fettig at gmail.com (John Fettig) Date: Tue, 5 Jun 2012 15:02:02 -0400 Subject: [petsc-users] Residual calculation in Richardson Message-ID: When using the defaults for Richardson, it calculates the (unpreconditioned) residual directly on line 138. However, this isn't made available to the convergence monitor because ksp->ops->buildresidual = KSPDefaultBuildResidual; So if you build the residual explicitly in the convergence monitor it must calculate it again. Could we do this instead? #undef __FUNCT__ #define __FUNCT__ "KSPBuildResidual_Richardson" PetscErrorCode KSPBuildResidual_Richardson(KSP ksp, Vec t, Vec v, Vec *V) { PetscErrorCode ierr; KSP_Richardson *ctx; PetscFunctionBegin; ctx = (KSP_Richardson *)ksp->data; if (ctx->selfscale) { KSPDefaultBuildResidual(ksp,t,v,V); } else { if (v) { ierr = VecCopy( ksp->work[0], v );CHKERRQ(ierr); if (V) *V = v; } else if (V) { *V = ksp->work[0]; } } PetscFunctionReturn(0); } John From samuelandjw at gmail.com Tue Jun 5 22:33:54 2012 From: samuelandjw at gmail.com (Wu Degang) Date: Wed, 06 Jun 2012 11:33:54 +0800 Subject: [petsc-users] linking to external library Message-ID: <4FCECFA2.1090101@ust.hk> Hi, In addition to petsc and slepc, I also need to link my program to external libraries such as GSL. I would like to modify the sample makefile in ${SLEPC_DIR}/src/eps/examples/tutorial/makefile rather than writing one from scratch. I modified the makefile so that ex1: ex1.o chkopts -${CLINKER} -o ex1 ex1.o ${SLEPC_LIB} ${RM} ex1.o becomes ex1: ex1.o chkopts -${CLINKER} -o ex1 ex1.o ${SLEPC_LIB} -lgsl -L/opt/local/lib ${RM} ex1.o But the linking process failed. The output: bash-3.2$ make ex1 /Users/wudegang/petsc-3.2-p7/arch-darwin-c-debug/bin/mpic++ -o ex1.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -I/Users/wudegang/petsc-3.2-p7/include -I/Users/wudegang/petsc-3.2-p7/arch-darwin-c-debug/include -I/opt/local/include -D__INSDIR__=src/eps/examples/tutorials/ -I/Users/wudegang/slepc-3.2-p5 -I/Users/wudegang/slepc-3.2-p5/arch-darwin-c-debug/include -I/Users/wudegang/slepc-3.2-p5/include ex1.c /Users/wudegang/petsc-3.2-p7/arch-darwin-c-debug/bin/mpic++ -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -o ex1 ex1.o -L/Users/wudegang/slepc-3.2-p5/arch-darwin-c-debug/lib -L/Users/wudegang/slepc-3.2-p5/arch-darwin-c-debug/lib -lslepc -L/Users/wudegang/petsc-3.2-p7/arch-darwin-c-debug/lib -lpetsc -L/usr/X11R6/lib -lX11 -lpthread -llapack -lblas -lmpi_cxx -lstdc++ -ldl -lgsl -L/opt/local/lib Undefined symbols for architecture x86_64: "_dgelqf", referenced from: IPCGSBiOrthogonalization(_p_IP*, int, _p_Vec**, _p_Vec**, _p_Vec*, double*, double*, double*)in libslepc.a(iporthog.c.o) "_dormlq", referenced from: IPCGSBiOrthogonalization(_p_IP*, int, _p_Vec**, _p_Vec**, _p_Vec*, double*, double*, double*)in libslepc.a(iporthog.c.o) "_dlaev2", referenced from: EPSSolve_TS_Power(_p_EPS*) in libslepc.a(power.c.o) EPSSolve_Power(_p_EPS*) in libslepc.a(power.c.o) "_dlanhs", referenced from: EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) "_dgetrf", referenced from: EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) EPSTranslateHarmonic(int, double*, int, double, double, double*, double*)in libslepc.a(ks-harm.c.o) MatLUFactor_SeqDense(_p_Mat*, _p_IS*, _p_IS*, MatFactorInfo const*)in libpetsc.a(dense.c.o) _KSPDGMRESComputeDeflationData_DGMRES in libpetsc.a(dgmres.c.o) _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) dvd_improvex_jd_proj_cuv(_dvdDashboard*, int, int, _p_Vec***, _p_Vec***, _p_Vec**, _p_Vec***, double**, double*, double*, double*, double*, int)in libslepc.a(dvd_improvex.c.o) "_dgetri", referenced from: EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) "_dlange", referenced from: EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) "_dsytrd", referenced from: ArrowTridFlip(int, int, double*, double*, double*, double*)in libslepc.a(ks-symm.c.o) "_dorgtr", referenced from: ArrowTridFlip(int, int, double*, double*, double*, double*)in libslepc.a(ks-symm.c.o) "_dsteqr", referenced from: ArrowTridFlip(int, int, double*, double*, double*, double*)in libslepc.a(ks-symm.c.o) "_dgetrs", referenced from: EPSTranslateHarmonic(int, double*, int, double, double, double*, double*)in libslepc.a(ks-harm.c.o) MatMatSolve_SeqDense(_p_Mat*, _p_Mat*, _p_Mat*)in libpetsc.a(dense.c.o) MatSolve_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveTranspose_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveTransposeAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) _KSPDGMRESApplyDeflation_DGMRES in libpetsc.a(dgmres.c.o) ... "_dgesvd", referenced from: EPSUpdateVectors(_p_EPS*, int, _p_Vec**, int, int, double*, int, double*, int)in libslepc.a(arnoldi.c.o) PCSetUp_SVD(_p_PC*) in libpetsc.a(svd.c.o) KSPComputeExtremeSingularValues_GMRES(_p_KSP*, double*, double*)in libpetsc.a(gmreig.c.o) "_dlasv2", referenced from: EPSCleanDenseSchur(int, int, double*, int, double*, int, double*, double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) "_dtgevc", referenced from: dvd_compute_eigenvectors(int, double*, int, double*, int, double*, int, double*, int, double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) "_dtrevc", referenced from: dvd_compute_eigenvectors(int, double*, int, double*, int, double*, int, double*, int, double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) EPSComputeVectors_Schur(_p_EPS*) in libslepc.a(default.c.o) QEPComputeVectors_Schur(_p_QEP*) in libslepc.a(qepdefault.c.o) DenseSelectedEvec(double*, int, double*, double*, int, PetscBool, int, double*)in libslepc.a(dense.c.o) "_dgeqrf", referenced from: SlepcDenseOrth(double*, int, int, int, double*, int, int*)in libslepc.a(dvd_blas.c.o) PCCreateTransferOp_ASA(__PC_ASA_level*, PetscBool) in libpetsc.a(asa.c.o) "_dorgqr", referenced from: SlepcDenseOrth(double*, int, int, int, double*, int, int*)in libslepc.a(dvd_blas.c.o) PCCreateTransferOp_ASA(__PC_ASA_level*, PetscBool) in libpetsc.a(asa.c.o) "_dpbtrf", referenced from: VecsOrthonormalize(_p_Vec**, int, double*, double*)in libslepc.a(dvd_blas.c.o) "_dgeevx", referenced from: EPSDenseNHEP(int, double*, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dstevr", referenced from: EPSDenseTridiagonal(int, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dtgexc", referenced from: EPSSortDenseSchurGeneralized(_p_EPS*, int, int, int, double*, double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dlamch", referenced from: EPSSortDenseSchurGeneralized(_p_EPS*, int, int, int, double*, double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dlag2", referenced from: EPSSortDenseSchurGeneralized(_p_EPS*, int, int, int, double*, double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dtrexc", referenced from: EPSSortDenseSchur(_p_EPS*, int, int, double*, int, double*, double*, double*)in libslepc.a(dense.c.o) QEPSortDenseSchur(_p_QEP*, int, int, double*, int, double*, double*, double*)in libslepc.a(qepdense.c.o) "_dhseqr", referenced from: EPSDenseSchur(int, int, double*, int, double*, double*, double*)in libslepc.a(dense.c.o) _KSPDGMRESComputeSchurForm_DGMRES in libpetsc.a(dgmres.c.o) "_dgehrd", referenced from: EPSDenseHessenberg(int, int, double*, int, double*)in libslepc.a(dense.c.o) "_dorghr", referenced from: EPSDenseHessenberg(int, int, double*, int, double*)in libslepc.a(dense.c.o) "_dsygvd", referenced from: EPSDenseGHEP(int, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dsyevr", referenced from: EPSDenseHEP(int, double*, int, double*, double*)in libslepc.a(dense.c.o) "_dggevx", referenced from: EPSDenseGNHEP(int, double*, double*, double*, double*, double*, double*)in libslepc.a(dense.c.o) "_dbdsdc", referenced from: SVDSolve_Lanczos(_p_SVD*) in libslepc.a(gklanczos.c.o) "_dgesdd", referenced from: SVDDense(int, int, double*, double*, double*, double*)in libslepc.a(svddense.c.o) "_dpotrf", referenced from: MatCholeskyFactor_SeqDense(_p_Mat*, _p_IS*, MatFactorInfo const*)in libpetsc.a(dense.c.o) KSPSolve_BCGSL(_p_KSP*) in libpetsc.a(bcgsl.c.o) "_dpotrs", referenced from: MatMatSolve_SeqDense(_p_Mat*, _p_Mat*, _p_Mat*)in libpetsc.a(dense.c.o) MatSolve_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveTranspose_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) MatSolveTransposeAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) KSPSolve_BCGSL(_p_KSP*) in libpetsc.a(bcgsl.c.o) "_dtrsen", referenced from: _KSPDGMRESComputeSchurForm_DGMRES in libpetsc.a(dgmres.c.o) "_dgerfs", referenced from: _KSPDGMRESApplyDeflation_DGMRES in libpetsc.a(dgmres.c.o) "_dgges", referenced from: _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) dvd_calcpairs_projeig_qz_gen(_dvdDashboard*) in libslepc.a(dvd_calcpairs.c.o) "_dtgsen", referenced from: _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) "_dstebz", referenced from: KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) "_dpttrf", referenced from: KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) "_dpttrs", referenced from: KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) "_dstein", referenced from: KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) "_dgeev", referenced from: KSPComputeEigenvaluesExplicitly(_p_KSP*, int, double*, double*)in libpetsc.a(eige.c.o) KSPComputeEigenvalues_GMRES(_p_KSP*, int, double*, double*, int*)in libpetsc.a(gmreig.c.o) ld: symbol(s) not found for architecture x86_64 collect2: ld returned 1 exit status make: [ex1] Error 1 (ignored) /bin/rm -f ex1.o Wu Degang From jedbrown at mcs.anl.gov Tue Jun 5 22:38:59 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 5 Jun 2012 22:38:59 -0500 Subject: [petsc-users] linking to external library In-Reply-To: <4FCECFA2.1090101@ust.hk> References: <4FCECFA2.1090101@ust.hk> Message-ID: 2012/6/5 Wu Degang > Hi, > > In addition to petsc and slepc, I also need to link my program to external > libraries such as GSL. I would like to modify the sample makefile in > ${SLEPC_DIR}/src/eps/examples/**tutorial/makefile rather than writing one > from scratch. I modified the makefile so that > > ex1: ex1.o chkopts > -${CLINKER} -o ex1 ex1.o ${SLEPC_LIB} > ${RM} ex1.o > > becomes > > ex1: ex1.o chkopts > -${CLINKER} -o ex1 ex1.o ${SLEPC_LIB} -lgsl -L/opt/local/lib > ${RM} ex1.o > > But the linking process failed. The output: > > bash-3.2$ make ex1 > /Users/wudegang/petsc-3.2-p7/**arch-darwin-c-debug/bin/mpic++ -o ex1.o -c > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g > -I/Users/wudegang/petsc-3.2-**p7/include -I/Users/wudegang/petsc-3.2-**p7/arch-darwin-c-debug/include > -I/opt/local/include -D__INSDIR__=src/eps/examples/**tutorials/ > -I/Users/wudegang/slepc-3.2-p5 -I/Users/wudegang/slepc-3.2-**p5/arch-darwin-c-debug/include > -I/Users/wudegang/slepc-3.2-**p5/include ex1.c > /Users/wudegang/petsc-3.2-p7/**arch-darwin-c-debug/bin/mpic++ > -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress > -Wl,-commons,use_dylibs -Wl,-search_paths_first > -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress > -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wall -Wwrite-strings > -Wno-strict-aliasing -Wno-unknown-pragmas -g -o ex1 ex1.o > -L/Users/wudegang/slepc-3.2-**p5/arch-darwin-c-debug/lib > -L/Users/wudegang/slepc-3.2-**p5/arch-darwin-c-debug/lib -lslepc > -L/Users/wudegang/petsc-3.2-**p7/arch-darwin-c-debug/lib -lpetsc > -L/usr/X11R6/lib -lX11 -lpthread -llapack -lblas -lmpi_cxx -lstdc++ -ldl > -lgsl -L/opt/local/lib > I'm going to guess that you have a functional liblapack and libblas at /usr/lib and a broken one (possibly installed by macports) in /opt/local/lib which gets picked up when the -L/opt/local/lib option is added. Fix is to remove the BLAS and LAPACK from in /opt/local/lib so that the one at /usr/lib can be used. > Undefined symbols for architecture x86_64: > "_dgelqf", referenced from: > IPCGSBiOrthogonalization(_p_**IP*, int, _p_Vec**, _p_Vec**, _p_Vec*, > double*, double*, double*)in libslepc.a(iporthog.c.o) > "_dormlq", referenced from: > IPCGSBiOrthogonalization(_p_**IP*, int, _p_Vec**, _p_Vec**, _p_Vec*, > double*, double*, double*)in libslepc.a(iporthog.c.o) > "_dlaev2", referenced from: > EPSSolve_TS_Power(_p_EPS*) in libslepc.a(power.c.o) > EPSSolve_Power(_p_EPS*) in libslepc.a(power.c.o) > "_dlanhs", referenced from: > EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) > "_dgetrf", referenced from: > EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) > EPSTranslateHarmonic(int, double*, int, double, double, double*, > double*)in libslepc.a(ks-harm.c.o) > MatLUFactor_SeqDense(_p_Mat*, _p_IS*, _p_IS*, MatFactorInfo const*)in > libpetsc.a(dense.c.o) > _**KSPDGMRESComputeDeflationData_**DGMRES in libpetsc.a(dgmres.c.o) > _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) > dvd_improvex_jd_proj_cuv(_**dvdDashboard*, int, int, _p_Vec***, > _p_Vec***, _p_Vec**, _p_Vec***, double**, double*, double*, double*, > double*, int)in libslepc.a(dvd_improvex.c.o) > "_dgetri", referenced from: > EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) > "_dlange", referenced from: > EPSHessCond(int, double*, int, double*)in libslepc.a(subspace.c.o) > "_dsytrd", referenced from: > ArrowTridFlip(int, int, double*, double*, double*, double*)in > libslepc.a(ks-symm.c.o) > "_dorgtr", referenced from: > ArrowTridFlip(int, int, double*, double*, double*, double*)in > libslepc.a(ks-symm.c.o) > "_dsteqr", referenced from: > ArrowTridFlip(int, int, double*, double*, double*, double*)in > libslepc.a(ks-symm.c.o) > "_dgetrs", referenced from: > EPSTranslateHarmonic(int, double*, int, double, double, double*, > double*)in libslepc.a(ks-harm.c.o) > MatMatSolve_SeqDense(_p_Mat*, _p_Mat*, _p_Mat*)in > libpetsc.a(dense.c.o) > MatSolve_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) > MatSolveTranspose_SeqDense(_p_**Mat*, _p_Vec*, _p_Vec*)in > libpetsc.a(dense.c.o) > MatSolveAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in > libpetsc.a(dense.c.o) > MatSolveTransposeAdd_SeqDense(**_p_Mat*, _p_Vec*, _p_Vec*, > _p_Vec*)in libpetsc.a(dense.c.o) > _KSPDGMRESApplyDeflation_**DGMRES in libpetsc.a(dgmres.c.o) > ... > "_dgesvd", referenced from: > EPSUpdateVectors(_p_EPS*, int, _p_Vec**, int, int, double*, int, > double*, int)in libslepc.a(arnoldi.c.o) > PCSetUp_SVD(_p_PC*) in libpetsc.a(svd.c.o) > KSPComputeExtremeSingularValue**s_GMRES(_p_KSP*, double*, double*)in > libpetsc.a(gmreig.c.o) > "_dlasv2", referenced from: > EPSCleanDenseSchur(int, int, double*, int, double*, int, double*, > double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) > "_dtgevc", referenced from: > dvd_compute_eigenvectors(int, double*, int, double*, int, double*, > int, double*, int, double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) > "_dtrevc", referenced from: > dvd_compute_eigenvectors(int, double*, int, double*, int, double*, > int, double*, int, double*, int, PetscBool)in libslepc.a(dvd_blas.c.o) > EPSComputeVectors_Schur(_p_**EPS*) in libslepc.a(default.c.o) > QEPComputeVectors_Schur(_p_**QEP*) in libslepc.a(qepdefault.c.o) > DenseSelectedEvec(double*, int, double*, double*, int, PetscBool, > int, double*)in libslepc.a(dense.c.o) > "_dgeqrf", referenced from: > SlepcDenseOrth(double*, int, int, int, double*, int, int*)in > libslepc.a(dvd_blas.c.o) > PCCreateTransferOp_ASA(__PC_**ASA_level*, PetscBool) in > libpetsc.a(asa.c.o) > "_dorgqr", referenced from: > SlepcDenseOrth(double*, int, int, int, double*, int, int*)in > libslepc.a(dvd_blas.c.o) > PCCreateTransferOp_ASA(__PC_**ASA_level*, PetscBool) in > libpetsc.a(asa.c.o) > "_dpbtrf", referenced from: > VecsOrthonormalize(_p_Vec**, int, double*, double*)in > libslepc.a(dvd_blas.c.o) > "_dgeevx", referenced from: > EPSDenseNHEP(int, double*, double*, double*, double*, double*)in > libslepc.a(dense.c.o) > "_dstevr", referenced from: > EPSDenseTridiagonal(int, double*, double*, double*, double*)in > libslepc.a(dense.c.o) > "_dtgexc", referenced from: > EPSSortDenseSchurGeneralized(_**p_EPS*, int, int, int, double*, > double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) > "_dlamch", referenced from: > EPSSortDenseSchurGeneralized(_**p_EPS*, int, int, int, double*, > double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) > "_dlag2", referenced from: > EPSSortDenseSchurGeneralized(_**p_EPS*, int, int, int, double*, > double*, int, double*, double*, double*, double*)in libslepc.a(dense.c.o) > "_dtrexc", referenced from: > EPSSortDenseSchur(_p_EPS*, int, int, double*, int, double*, double*, > double*)in libslepc.a(dense.c.o) > QEPSortDenseSchur(_p_QEP*, int, int, double*, int, double*, double*, > double*)in libslepc.a(qepdense.c.o) > "_dhseqr", referenced from: > EPSDenseSchur(int, int, double*, int, double*, double*, double*)in > libslepc.a(dense.c.o) > _KSPDGMRESComputeSchurForm_**DGMRES in libpetsc.a(dgmres.c.o) > "_dgehrd", referenced from: > EPSDenseHessenberg(int, int, double*, int, double*)in > libslepc.a(dense.c.o) > "_dorghr", referenced from: > EPSDenseHessenberg(int, int, double*, int, double*)in > libslepc.a(dense.c.o) > "_dsygvd", referenced from: > EPSDenseGHEP(int, double*, double*, double*, double*)in > libslepc.a(dense.c.o) > "_dsyevr", referenced from: > EPSDenseHEP(int, double*, int, double*, double*)in > libslepc.a(dense.c.o) > "_dggevx", referenced from: > EPSDenseGNHEP(int, double*, double*, double*, double*, double*, > double*)in libslepc.a(dense.c.o) > "_dbdsdc", referenced from: > SVDSolve_Lanczos(_p_SVD*) in libslepc.a(gklanczos.c.o) > "_dgesdd", referenced from: > SVDDense(int, int, double*, double*, double*, double*)in > libslepc.a(svddense.c.o) > "_dpotrf", referenced from: > MatCholeskyFactor_SeqDense(_p_**Mat*, _p_IS*, MatFactorInfo > const*)in libpetsc.a(dense.c.o) > KSPSolve_BCGSL(_p_KSP*) in libpetsc.a(bcgsl.c.o) > "_dpotrs", referenced from: > MatMatSolve_SeqDense(_p_Mat*, _p_Mat*, _p_Mat*)in > libpetsc.a(dense.c.o) > MatSolve_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*)in libpetsc.a(dense.c.o) > MatSolveTranspose_SeqDense(_p_**Mat*, _p_Vec*, _p_Vec*)in > libpetsc.a(dense.c.o) > MatSolveAdd_SeqDense(_p_Mat*, _p_Vec*, _p_Vec*, _p_Vec*)in > libpetsc.a(dense.c.o) > MatSolveTransposeAdd_SeqDense(**_p_Mat*, _p_Vec*, _p_Vec*, > _p_Vec*)in libpetsc.a(dense.c.o) > KSPSolve_BCGSL(_p_KSP*) in libpetsc.a(bcgsl.c.o) > "_dtrsen", referenced from: > _KSPDGMRESComputeSchurForm_**DGMRES in libpetsc.a(dgmres.c.o) > "_dgerfs", referenced from: > _KSPDGMRESApplyDeflation_**DGMRES in libpetsc.a(dgmres.c.o) > "_dgges", referenced from: > _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) > dvd_calcpairs_projeig_qz_gen(_**dvdDashboard*) in > libslepc.a(dvd_calcpairs.c.o) > "_dtgsen", referenced from: > _KSPDGMRESImproveEig_DGMRES in libpetsc.a(dgmres.c.o) > "_dstebz", referenced from: > KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) > "_dpttrf", referenced from: > KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) > "_dpttrs", referenced from: > KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) > "_dstein", referenced from: > KSPSolve_GLTR(_p_KSP*) in libpetsc.a(gltr.c.o) > "_dgeev", referenced from: > KSPComputeEigenvaluesExplicitl**y(_p_KSP*, int, double*, double*)in > libpetsc.a(eige.c.o) > KSPComputeEigenvalues_GMRES(_**p_KSP*, int, double*, double*, > int*)in libpetsc.a(gmreig.c.o) > ld: symbol(s) not found for architecture x86_64 > collect2: ld returned 1 exit status > make: [ex1] Error 1 (ignored) > /bin/rm -f ex1.o > > Wu Degang > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Tue Jun 5 22:54:57 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Tue, 5 Jun 2012 21:54:57 -0600 Subject: [petsc-users] segfault running src/dm/examples/tutorials/ex11f90.F Message-ID: Dear list, I run into a segfault when running src/dm/examples/tutorials/ex11f90.F Intel Fortran Compiler 12.1.3; 1.6 OpenMPI; Ubuntu 12.04 other examples compiles and runs well. It compiles without any warnings or errors; however it cannot run. Any ideas? Error message are attached. Thank you! Panruo Wu $ mpirun -np 1 ex11f90 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ex11f90 on a opmi_inte named avahome by pwu Tue Jun 5 21:44:11 2012 [0]PETSC ERROR: Libraries linked from /home/pwu/Downloads/petsc-3.2-p7/opmi_intel_dbg/lib [0]PETSC ERROR: Configure run at Tue Jun 5 10:02:00 2012 [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64 --download-umfpack --download-superlu --with-x=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 22293 on node avahome exiting improperly. There are two reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at mcs.anl.gov Tue Jun 5 23:12:46 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Tue, 5 Jun 2012 23:12:46 -0500 Subject: [petsc-users] linking to external library In-Reply-To: References: <4FCECFA2.1090101@ust.hk> Message-ID: > I'm going to guess that you have a functional liblapack and libblas at > /usr/lib and a broken one (possibly installed by macports) in /opt/local/lib > which gets picked up when the -L/opt/local/lib option is added. > > Fix is to remove the BLAS and LAPACK from in /opt/local/lib so that the one > at /usr/lib can be used. It's not so much that ATLAS is broken but that random symbols are getting picked up and linked. But yes, Jed is correct; you really need to uninstall ATLAS from MacPorts (it's usually installed from the numpy or scipy port): $ sudo port -v uninstall --follow-dependents atlas Then reinstall whichever ports you need but without atlas: $ sudo port -v install py27-numpy -atlas Be warned with the above command line since I'm just giving an example and don't actually know which ports or variants that you need. From samuelandjw at gmail.com Wed Jun 6 01:25:56 2012 From: samuelandjw at gmail.com (Wu Degang) Date: Wed, 06 Jun 2012 14:25:56 +0800 Subject: [petsc-users] linking to external library Message-ID: <4FCEF7F4.4090701@ust.hk> >> I'm going to guess that you have a functional liblapack and libblas at >> /usr/lib and a broken one (possibly installed by macports) in /opt/local/lib >> which gets picked up when the -L/opt/local/lib option is added. >> >> Fix is to remove the BLAS and LAPACK from in /opt/local/lib so that the one >> at /usr/lib can be used. > It's not so much that ATLAS is broken but that random symbols are > getting picked up and linked. But yes, Jed is correct; you really need > to uninstall ATLAS from MacPorts (it's usually installed from the > numpy or scipy port): > > $ sudo port -v uninstall --follow-dependents atlas > > Then reinstall whichever ports you need but without atlas: > > $ sudo port -v install py27-numpy -atlas > > Be warned with the above command line since I'm just giving an example > and don't actually know which ports or variants that you need. Thanks to Sean and Jed. Uninstalling atlas does the job. From C.Klaij at marin.nl Wed Jun 6 02:17:14 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Wed, 6 Jun 2012 07:17:14 +0000 Subject: [petsc-users] GMRES pc right fails while pc left succeeds Message-ID: > In theory if you use petsc-3.3 (release any day) switching to use > the PCFIELDSPLIT solver (with appropriate options) should be able > to reproduce your algorithm (I think) with very little coding on > your part. This would make it easier for us to help you understand > the issue since we know exactly what PCFIELDSPLIT does and how it > is suppose to work and the -ksp_view option (or -snes_view) tells > exactly the algorithm being used. > > Barry Yes, I've been thinking about that, it should be possible without too much effort. At this point I just don't have any time but I'll keep it in mind. Thanks to Jed and to you for your answers! dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From agrayver at gfz-potsdam.de Wed Jun 6 05:22:03 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 06 Jun 2012 12:22:03 +0200 Subject: [petsc-users] MatSetValues and values cache Message-ID: <4FCF2F4B.9010806@gfz-potsdam.de> Hi, In my program I assemble huge matrix (~500 millions of double complex nnz) and all MatSetValues calls are done before MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY). I'm wondering if it makes sense to use MatAssemblyBegin(A, MAT_FLUSH_ASSEMBLY) in between? In docs it is said: >> MatSetValues () generally caches the values. How does the caching work actually? Is this additionally allocated memory (dynamic or not)? Thanks. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 6 06:06:52 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 6 Jun 2012 06:06:52 -0500 Subject: [petsc-users] MatSetValues and values cache In-Reply-To: <4FCF2F4B.9010806@gfz-potsdam.de> References: <4FCF2F4B.9010806@gfz-potsdam.de> Message-ID: On Wed, Jun 6, 2012 at 5:22 AM, Alexander Grayver wrote: > In my program I assemble huge matrix (~500 millions of double complex nnz) > and all MatSetValues calls are done before MatAssemblyBegin(A, > MAT_FINAL_ASSEMBLY). > I'm wondering if it makes sense to use MatAssemblyBegin(A, > MAT_FLUSH_ASSEMBLY) in between? In docs it is said: > >> MatSetValues() > generally caches the values. > > How does the caching work actually? Is this additionally allocated memory > (dynamic or not)? > Only entries that are generated on a different process than they need to be stored will be cached. If you generate most entries on the correct process (the owned rows for MPI*AIJ matrices), there is no need to flush. You can also look at -log_summary to see if an inordinate amount of time is spent in assembly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 6 06:41:42 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 06 Jun 2012 13:41:42 +0200 Subject: [petsc-users] MatSetValues and values cache In-Reply-To: References: <4FCF2F4B.9010806@gfz-potsdam.de> Message-ID: <4FCF41F6.1040205@gfz-potsdam.de> On 06.06.2012 13:06, Jed Brown wrote: > On Wed, Jun 6, 2012 at 5:22 AM, Alexander Grayver > > wrote: > > In my program I assemble huge matrix (~500 millions of double > complex nnz) and all MatSetValues calls are done before > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY). > I'm wondering if it makes sense to use MatAssemblyBegin(A, > MAT_FLUSH_ASSEMBLY) in between? In docs it is said: > >> MatSetValues > () > generally caches the values. > > How does the caching work actually? Is this additionally allocated > memory (dynamic or not)? > > > Only entries that are generated on a different process than they need > to be stored will be cached. If you generate most entries on the > correct process (the owned rows for MPI*AIJ matrices), there is no > need to flush. Jed, Makes sense. I should have thought about that. I have off process entries, although not many. Thanks. > > You can also look at -log_summary to see if an inordinate amount of > time is spent in assembly. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 6 07:27:21 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 06 Jun 2012 14:27:21 +0200 Subject: [petsc-users] Error message "Row/Column is too large" Message-ID: <4FCF4CA9.1070400@gfz-potsdam.de> Hi, In case of SeqDense matrix the message if very usefull since prints the maximum and actual indices: MatSetValues_SeqDense() line 750 in /lib/petsc-dev1/src/mat/impls/dense/seq/dense.c if (indexn[j] >= A->cmap->n) SETERRQ2(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE,"Column too large: col %D max %D",indexn[j],A->cmap->n-1); For MPIDense it is not the case: MatSetValues_MPIDense() line 135 in /lib/petsc-dev/src/mat/impls/dense/mpi/mpidense.c if (idxm[i] >= mat->rmap->N) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE,"Row too large"); Would it be possible to have the same message for MPIDense as well? Thanks. -- Regards, Alexander From bsmith at mcs.anl.gov Wed Jun 6 11:07:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 6 Jun 2012 11:07:07 -0500 Subject: [petsc-users] Error message "Row/Column is too large" In-Reply-To: <4FCF4CA9.1070400@gfz-potsdam.de> References: <4FCF4CA9.1070400@gfz-potsdam.de> Message-ID: <82F442D2-F822-4557-86D5-7B87D55EEC45@mcs.anl.gov> On Jun 6, 2012, at 7:27 AM, Alexander Grayver wrote: > Hi, > > In case of SeqDense matrix the message if very usefull since prints the maximum and actual indices: > > MatSetValues_SeqDense() line 750 in /lib/petsc-dev1/src/mat/impls/dense/seq/dense.c > if (indexn[j] >= A->cmap->n) SETERRQ2(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE,"Column too large: col %D max %D",indexn[j],A->cmap->n-1); > > For MPIDense it is not the case: > > MatSetValues_MPIDense() line 135 in /lib/petsc-dev/src/mat/impls/dense/mpi/mpidense.c > if (idxm[i] >= mat->rmap->N) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_OUTOFRANGE,"Row too large"); We are completely replacing the MPIDense matrices this summer (hopefully) and will provide this information as well as much more functionality (hopefully). Barry > > Would it be possible to have the same message for MPIDense as well? > Thanks. > > -- > Regards, > Alexander > From jedbrown at mcs.anl.gov Wed Jun 6 13:20:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 6 Jun 2012 13:20:57 -0500 Subject: [petsc-users] Residual calculation in Richardson In-Reply-To: References: Message-ID: I don't have a problem with a KSPBuildResidual_Richardson, but what about also removing the part that forms the residual at the end of each iteration so that it is skipped if KSP_NORM_NONE? On Tue, Jun 5, 2012 at 2:02 PM, John Fettig wrote: > When using the defaults for Richardson, it calculates the > (unpreconditioned) residual directly on line 138. However, this isn't > made available to the convergence monitor because > ksp->ops->buildresidual = KSPDefaultBuildResidual; > > So if you build the residual explicitly in the convergence monitor it > must calculate it again. Could we do this instead? > > #undef __FUNCT__ > #define __FUNCT__ "KSPBuildResidual_Richardson" > PetscErrorCode KSPBuildResidual_Richardson(KSP ksp, Vec t, Vec v, Vec *V) > { > PetscErrorCode ierr; > KSP_Richardson *ctx; > > PetscFunctionBegin; > ctx = (KSP_Richardson *)ksp->data; > if (ctx->selfscale) { > KSPDefaultBuildResidual(ksp,t,v,V); > } else { > if (v) { > ierr = VecCopy( ksp->work[0], v );CHKERRQ(ierr); > if (V) *V = v; > } else if (V) { > *V = ksp->work[0]; > } > } > PetscFunctionReturn(0); > } > > John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Wed Jun 6 15:04:07 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 06 Jun 2012 22:04:07 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> Message-ID: <4FCFB7B7.1090304@gmail.com> Hi, I have used 3 KSP, 2 to solve momentum eqns and 1 for the multigrid. I have used call KSPSetOptionsPrefix(ksp,"mg_",ierr) for the multigrid. I run with : -log_summary -mg_ksp_view so as to single out the multigrid ksp, but I'm not sure if it's really working... Here's the output: ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.2 named n12-50 with 4 processors, by wtay Wed Jun 6 21:57:33 2012 Using Petsc Development HG revision: c76fb3cac2a4ad0dfc9436df80f678898c867e86 HG Date: Thu May 31 00:33:26 2012 -0500 Max Max/Min Avg Total Time (sec): 1.064e+01 1.00000 1.064e+01 Objects: 2.700e+01 1.00000 2.700e+01 Flops: 4.756e+08 1.00811 4.744e+08 1.897e+09 Flops/sec: 4.468e+07 1.00811 4.457e+07 1.783e+08 MPI Messages: 4.080e+02 2.00000 3.060e+02 1.224e+03 MPI Message Lengths: 2.328e+06 2.00000 5.706e+03 6.984e+06 MPI Reductions: 8.750e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.0644e+01 100.0% 1.8975e+09 100.0% 1.224e+03 100.0% 5.706e+03 100.0% 8.740e+02 99.9% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 202 1.0 5.5096e-01 1.0 1.38e+08 1.0 1.2e+03 5.7e+03 0.0e+00 5 29 99100 0 5 29 99100 0 998 MatSolve 252 1.0 6.9136e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 6 36 0 0 0 6 36 0 0 0 986 MatLUFactorNum 50 1.0 4.6002e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00 4 15 0 0 0 4 15 0 0 0 634 MatILUFactorSym 1 1.0 9.5899e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 50 1.0 1.6270e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 11 0 0 0 0 11 0 MatAssemblyEnd 50 1.0 1.0896e-01 1.0 0.00e+00 0.0 1.2e+01 1.4e+03 8.0e+00 1 0 1 0 1 1 0 1 0 1 0 MatGetRowIJ 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 7.2002e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 100 1.0 2.9130e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 50 1.0 2.0737e+00 1.0 4.76e+08 1.0 1.2e+03 5.7e+03 4.6e+02 19100 99100 52 19100 99100 53 915 VecDot 202 1.0 7.3588e-02 1.1 1.63e+07 1.0 0.0e+00 0.0e+00 2.0e+02 1 3 0 0 23 1 3 0 0 23 885 VecDotNorm2 101 1.0 3.9155e-02 1.7 1.63e+07 1.0 0.0e+00 0.0e+00 1.0e+02 0 3 0 0 12 0 3 0 0 12 1664 VecNorm 151 1.0 5.8769e-02 1.7 1.22e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 3 0 0 17 0 3 0 0 17 829 VecCopy 100 1.0 2.3459e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 403 1.0 5.9994e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 202 1.0 6.6376e-02 1.0 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 1963 VecWAXPY 202 1.0 6.9311e-02 1.0 1.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 940 VecAssemblyBegin 100 1.0 4.0355e-0214.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+02 0 0 0 0 34 0 0 0 0 34 0 VecAssemblyEnd 100 1.0 5.0378e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 202 1.0 6.2275e-03 1.5 0.00e+00 0.0 1.2e+03 5.7e+03 0.0e+00 0 0 99100 0 0 0 99100 0 0 VecScatterEnd 202 1.0 2.0878e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 100 1.0 4.7225e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 5.0e+00 4 15 0 0 1 4 15 0 0 1 617 PCSetUpOnBlocks 50 1.0 4.7191e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 3.0e+00 4 15 0 0 0 4 15 0 0 0 618 PCApply 252 1.0 7.3425e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 7 36 0 0 0 7 36 0 0 0 928 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 4 4 16900896 0 Krylov Solver 2 2 2168 0 Vector 12 12 2604080 0 Vector Scatter 1 1 1060 0 Index Set 5 5 167904 0 Preconditioner 2 2 1800 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.09673e-06 Average time for MPI_Barrier(): 4.00543e-06 Average time for zero size MPI_Send(): 1.22786e-05 #PETSc Option Table entries: -log_summary -mg_ksp_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Thu May 31 09:53:43 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Thu May 31 09:53:43 2012 on hpc12 Machine characteristics: Linux-2.6.32-220.2.1.el6.x86_64-x86_64-with-centos-6.2-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.2-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -Qoption,cpp,--extended_float_type -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lHYPRE -lmpi_cxx -Wl,-rpath,/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lstdc++ -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl -L/opt/openmpi-1.5.3/lib -lmpi -lnsl -lutil -L/opt/intelcpro-11.1.059/lib/intel64 -limf -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- Yours sincerely, TAY wee-beng On 5/6/2012 1:34 AM, Barry Smith wrote: > Also run with -ksp_view to see exasctly what solver options it is using. For example the number of levels, smoother on each level etc. My guess is that the below is running on one level (because I don't see you supplying options to control the number of levels etc). > > Barry > > On Jun 4, 2012, at 4:15 PM, Jed Brown wrote: > >> Always send -log_summary when asking about performance. >> >> On Mon, Jun 4, 2012 at 4:11 PM, TAY wee-beng wrote: >> Hi, >> >> I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. ex29 to Fortran and then added into my code to solve the Poisson equation. >> >> The main subroutines are: >> >> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >> >> call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) >> call DMSetFunction(da,ComputeRHS,ierr) >> call DMSetJacobian(da,ComputeMatrix,ierr) >> call KSPSetDM(ksp,da,ierr) >> >> call KSPSetFromOptions(ksp,ierr) >> call KSPSetUp(ksp,ierr) >> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) >> call KSPGetSolution(ksp,x,ierr) >> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) >> call KSPDestroy(ksp,ierr) >> call DMDestroy(da,ierr) >> call PetscFinalize(ierr) >> >> >> Since the LHS matrix doesn't change, I only set up at the 1st time step, thereafter I only called ComputeRHS every time step. >> >> I was using HYPRE's geometric multigrid and the speed was much faster. >> >> What other options can I tweak to improve the speed? Or should I call the subroutines above at every timestep? >> >> Thanks! >> >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> From bsmith at mcs.anl.gov Wed Jun 6 15:31:05 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 6 Jun 2012 15:31:05 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FCFB7B7.1090304@gmail.com> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> Message-ID: <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> On Jun 6, 2012, at 3:04 PM, TAY wee-beng wrote: > Hi, > > I have used 3 KSP, 2 to solve momentum eqns and 1 for the multigrid. I have used > > call KSPSetOptionsPrefix(ksp,"mg_",ierr) for the multigrid. > > I run with : > > -log_summary -mg_ksp_view so as to single out the multigrid ksp, but I'm not sure if it's really working... Are you sure the KSPSetOptionsPrefix() is called before the KSPSetFromOptions()? It appears the prefix is not being set correctly. From the limited data below it is running multigrid with one level (hence you cannot expect great performance). You need to at least provide MG with a bit more information like how many levels you would like it to use. Barry > > Here's the output: > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > ./a.out on a petsc-3.2 named n12-50 with 4 processors, by wtay Wed Jun 6 21:57:33 2012 > Using Petsc Development HG revision: c76fb3cac2a4ad0dfc9436df80f678898c867e86 HG Date: Thu May 31 00:33:26 2012 -0500 > > Max Max/Min Avg Total > Time (sec): 1.064e+01 1.00000 1.064e+01 > Objects: 2.700e+01 1.00000 2.700e+01 > Flops: 4.756e+08 1.00811 4.744e+08 1.897e+09 > Flops/sec: 4.468e+07 1.00811 4.457e+07 1.783e+08 > MPI Messages: 4.080e+02 2.00000 3.060e+02 1.224e+03 > MPI Message Lengths: 2.328e+06 2.00000 5.706e+03 6.984e+06 > MPI Reductions: 8.750e+02 1.00000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flops > and VecAXPY() for complex vectors of length N --> 8N flops > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total Avg %Total counts %Total > 0: Main Stage: 1.0644e+01 100.0% 1.8975e+09 100.0% 1.224e+03 100.0% 5.706e+03 100.0% 8.740e+02 99.9% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flops: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > Avg. len: average message length > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %f - percent flops in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > --- Event Stage 0: Main Stage > > MatMult 202 1.0 5.5096e-01 1.0 1.38e+08 1.0 1.2e+03 5.7e+03 0.0e+00 5 29 99100 0 5 29 99100 0 998 > MatSolve 252 1.0 6.9136e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 6 36 0 0 0 6 36 0 0 0 986 > MatLUFactorNum 50 1.0 4.6002e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00 4 15 0 0 0 4 15 0 0 0 634 > MatILUFactorSym 1 1.0 9.5899e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 50 1.0 1.6270e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 11 0 0 0 0 11 0 > MatAssemblyEnd 50 1.0 1.0896e-01 1.0 0.00e+00 0.0 1.2e+01 1.4e+03 8.0e+00 1 0 1 0 1 1 0 1 0 1 0 > MatGetRowIJ 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 1 1.0 7.2002e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSetUp 100 1.0 2.9130e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 50 1.0 2.0737e+00 1.0 4.76e+08 1.0 1.2e+03 5.7e+03 4.6e+02 19100 99100 52 19100 99100 53 915 > VecDot 202 1.0 7.3588e-02 1.1 1.63e+07 1.0 0.0e+00 0.0e+00 2.0e+02 1 3 0 0 23 1 3 0 0 23 885 > VecDotNorm2 101 1.0 3.9155e-02 1.7 1.63e+07 1.0 0.0e+00 0.0e+00 1.0e+02 0 3 0 0 12 0 3 0 0 12 1664 > VecNorm 151 1.0 5.8769e-02 1.7 1.22e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 3 0 0 17 0 3 0 0 17 829 > VecCopy 100 1.0 2.3459e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 403 1.0 5.9994e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > VecAXPBYCZ 202 1.0 6.6376e-02 1.0 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 1963 > VecWAXPY 202 1.0 6.9311e-02 1.0 1.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 940 > VecAssemblyBegin 100 1.0 4.0355e-0214.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+02 0 0 0 0 34 0 0 0 0 34 0 > VecAssemblyEnd 100 1.0 5.0378e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 202 1.0 6.2275e-03 1.5 0.00e+00 0.0 1.2e+03 5.7e+03 0.0e+00 0 0 99100 0 0 0 99100 0 0 > VecScatterEnd 202 1.0 2.0878e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 100 1.0 4.7225e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 5.0e+00 4 15 0 0 1 4 15 0 0 1 617 > PCSetUpOnBlocks 50 1.0 4.7191e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 3.0e+00 4 15 0 0 0 4 15 0 0 0 618 > PCApply 252 1.0 7.3425e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 7 36 0 0 0 7 36 0 0 0 928 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Matrix 4 4 16900896 0 > Krylov Solver 2 2 2168 0 > Vector 12 12 2604080 0 > Vector Scatter 1 1 1060 0 > Index Set 5 5 167904 0 > Preconditioner 2 2 1800 0 > Viewer 1 0 0 0 > ======================================================================================================================== > Average time to get PetscTime(): 1.09673e-06 > Average time for MPI_Barrier(): 4.00543e-06 > Average time for zero size MPI_Send(): 1.22786e-05 > #PETSc Option Table entries: > > -log_summary > -mg_ksp_view > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure run at: Thu May 31 09:53:43 2012 > Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries > ----------------------------------------- > Libraries compiled on Thu May 31 09:53:43 2012 on hpc12 > Machine characteristics: Linux-2.6.32-220.2.1.el6.x86_64-x86_64-with-centos-6.2-Final > Using PETSc directory: /home/wtay/Codes/petsc-dev > Using PETSc arch: petsc-3.2-dev_shared_rel > ----------------------------------------- > > Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -Qoption,cpp,--extended_float_type -O3 ${COPTFLAGS} ${CFLAGS} > Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} > ----------------------------------------- > > Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/opt/openmpi-1.5.3/include > ----------------------------------------- > > Using C linker: /opt/openmpi-1.5.3/bin/mpicc > Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 > Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lHYPRE -lmpi_cxx -Wl,-rpath,/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lstdc++ -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl -L/opt/openmpi-1.5.3/lib -lmpi -lnsl -lutil -L/opt/intelcpro-11.1.059/lib/intel64 -limf -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl > ----------------------------------------- > > Yours sincerely, > > TAY wee-beng > > > On 5/6/2012 1:34 AM, Barry Smith wrote: >> Also run with -ksp_view to see exasctly what solver options it is using. For example the number of levels, smoother on each level etc. My guess is that the below is running on one level (because I don't see you supplying options to control the number of levels etc). >> >> Barry >> >> On Jun 4, 2012, at 4:15 PM, Jed Brown wrote: >> >>> Always send -log_summary when asking about performance. >>> >>> On Mon, Jun 4, 2012 at 4:11 PM, TAY wee-beng wrote: >>> Hi, >>> >>> I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. ex29 to Fortran and then added into my code to solve the Poisson equation. >>> >>> The main subroutines are: >>> >>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>> >>> call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) >>> call DMSetFunction(da,ComputeRHS,ierr) >>> call DMSetJacobian(da,ComputeMatrix,ierr) >>> call KSPSetDM(ksp,da,ierr) >>> >>> call KSPSetFromOptions(ksp,ierr) >>> call KSPSetUp(ksp,ierr) >>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) >>> call KSPGetSolution(ksp,x,ierr) >>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) >>> call KSPDestroy(ksp,ierr) >>> call DMDestroy(da,ierr) >>> call PetscFinalize(ierr) >>> >>> >>> Since the LHS matrix doesn't change, I only set up at the 1st time step, thereafter I only called ComputeRHS every time step. >>> >>> I was using HYPRE's geometric multigrid and the speed was much faster. >>> >>> What other options can I tweak to improve the speed? Or should I call the subroutines above at every timestep? >>> >>> Thanks! >>> >>> >>> -- >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> From zonexo at gmail.com Wed Jun 6 16:21:40 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 06 Jun 2012 23:21:40 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> Message-ID: <4FCFC9E4.3050301@gmail.com> On 6/6/2012 10:31 PM, Barry Smith wrote: > On Jun 6, 2012, at 3:04 PM, TAY wee-beng wrote: > >> Hi, >> >> I have used 3 KSP, 2 to solve momentum eqns and 1 for the multigrid. I have used >> >> call KSPSetOptionsPrefix(ksp,"mg_",ierr) for the multigrid. >> >> I run with : >> >> -log_summary -mg_ksp_view so as to single out the multigrid ksp, but I'm not sure if it's really working... > Are you sure the KSPSetOptionsPrefix() is called before the KSPSetFromOptions()? It appears the prefix is not being set correctly. > > From the limited data below it is running multigrid with one level (hence you cannot expect great performance). You need to at least provide MG with a bit more information like how many levels you would like it to use. > > Barry Ya, I called KSPSetOptionsPrefix after KSPSetFromOptions. I've changed it. I've included the input below. Btw, in my code, I followed the example in ex29.c which uses: *call KSPCreate(MPI_COMM_WORLD,ksp,ierr) call DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,1,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) call DMSetFunction(da,ComputeRHS,ierr) call DMSetJacobian(da,ComputeMatrix,ierr) call KSPSetDM(ksp,da,ierr) call KSPSetOptionsPrefix(ksp,"mg_",ierr) call KSPSetFromOptions(ksp,ierr) tol=1.e-5 call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,ierr) call KSPSetUp(ksp,ierr) call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) * I just read the manual and it says for multigrid, I have to use: KSPGetPC(KSP ksp,PC *pc); PCSetType(PC pc,PCMG); PCMGSetLevels(pc,int levels,MPI Comm *comms) I included after after KSPCreate with: *call KSPGetPC(ksp,pc,ierr) call PCSetType(pc_uv,PCMG,ierr) mg_lvl = 2 call PCMGSetLevels(pc,mg_lvl,MPI_COMM_WORLD,ierr)* However, I get the error: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range after calling *PCMGSetLevels* What's the problem? Is there any examples which I can follow? Thanks! *---------------------------------------------- PETSc Performance Summary: ----------------------------------------------* ./a.out on a petsc-3.2 named n12-50 with 4 processors, by wtay Wed Jun 6 22:45:37 2012 Using Petsc Development HG revision: c76fb3cac2a4ad0dfc9436df80f678898c867e86 HG Date: Thu May 31 00:33:26 2012 -0500 Max Max/Min Avg Total Time (sec): 1.062e+01 1.00001 1.062e+01 Objects: 2.700e+01 1.00000 2.700e+01 Flops: 4.756e+08 1.00811 4.744e+08 1.897e+09 Flops/sec: 4.477e+07 1.00811 4.466e+07 1.786e+08 MPI Messages: 4.080e+02 2.00000 3.060e+02 1.224e+03 MPI Message Lengths: 2.328e+06 2.00000 5.706e+03 6.984e+06 MPI Reductions: 8.750e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.0623e+01 100.0% 1.8975e+09 100.0% 1.224e+03 100.0% 5.706e+03 100.0% 8.740e+02 99.9% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 202 1.0 5.5212e-01 1.0 1.38e+08 1.0 1.2e+03 5.7e+03 0.0e+00 5 29 99100 0 5 29 99100 0 996 MatSolve 252 1.0 6.8899e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 6 36 0 0 0 6 36 0 0 0 989 MatLUFactorNum 50 1.0 4.5529e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00 4 15 0 0 0 4 15 0 0 0 640 MatILUFactorSym 1 1.0 9.7420e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 50 1.0 1.7412e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 11 0 0 0 0 11 0 MatAssemblyEnd 50 1.0 1.0649e-01 1.0 0.00e+00 0.0 1.2e+01 1.4e+03 8.0e+00 1 0 1 0 1 1 0 1 0 1 0 MatGetRowIJ 1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 7.0190e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 100 1.0 2.9013e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 50 1.0 2.0438e+00 1.0 4.76e+08 1.0 1.2e+03 5.7e+03 4.6e+02 19100 99100 52 19100 99100 53 928 VecDot 202 1.0 6.9886e-02 1.4 1.63e+07 1.0 0.0e+00 0.0e+00 2.0e+02 1 3 0 0 23 1 3 0 0 23 932 VecDotNorm2 101 1.0 4.0677e-02 2.1 1.63e+07 1.0 0.0e+00 0.0e+00 1.0e+02 0 3 0 0 12 0 3 0 0 12 1602 VecNorm 151 1.0 3.5888e-02 1.4 1.22e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 3 0 0 17 0 3 0 0 17 1357 VecCopy 100 1.0 2.2957e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 403 1.0 6.1034e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 202 1.0 6.6927e-02 1.0 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 1947 VecWAXPY 202 1.0 7.0219e-02 1.1 1.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 928 VecAssemblyBegin 100 1.0 4.0812e-0213.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+02 0 0 0 0 34 0 0 0 0 34 0 VecAssemblyEnd 100 1.0 4.8542e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 202 1.0 7.2360e-03 1.7 0.00e+00 0.0 1.2e+03 5.7e+03 0.0e+00 0 0 99100 0 0 0 99100 0 0 VecScatterEnd 202 1.0 2.7255e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 100 1.0 4.6843e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 5.0e+00 4 15 0 0 1 4 15 0 0 1 622 PCSetUpOnBlocks 50 1.0 4.6814e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 3.0e+00 4 15 0 0 0 4 15 0 0 0 623 PCApply 252 1.0 7.3618e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 7 36 0 0 0 7 36 0 0 0 926 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 4 4 16900896 0 Krylov Solver 2 2 2168 0 Vector 12 12 2604080 0 Vector Scatter 1 1 1060 0 Index Set 5 5 167904 0 Preconditioner 2 2 1800 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.19209e-06 Average time for MPI_Barrier(): 6.62804e-06 Average time for zero size MPI_Send(): 2.02656e-05 #PETSc Option Table entries: -log_summary -mg_ksp_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Thu May 31 09:53:43 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Thu May 31 09:53:43 2012 on hpc12 Machine characteristics: Linux-2.6.32-220.2.1.el6.x86_64-x86_64-with-centos-6.2-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.2-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -Qoption,cpp,--extended_float_type -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lHYPRE -lmpi_cxx -Wl,-rpath,/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lstdc++ -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl -L/opt/openmpi-1.5.3/lib -lmpi -lnsl -lutil -L/opt/intelcpro-11.1.059/lib/intel64 -limf -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl >> Here's the output: >> >> ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- >> >> ./a.out on a petsc-3.2 named n12-50 with 4 processors, by wtay Wed Jun 6 21:57:33 2012 >> Using Petsc Development HG revision: c76fb3cac2a4ad0dfc9436df80f678898c867e86 HG Date: Thu May 31 00:33:26 2012 -0500 >> >> Max Max/Min Avg Total >> Time (sec): 1.064e+01 1.00000 1.064e+01 >> Objects: 2.700e+01 1.00000 2.700e+01 >> Flops: 4.756e+08 1.00811 4.744e+08 1.897e+09 >> Flops/sec: 4.468e+07 1.00811 4.457e+07 1.783e+08 >> MPI Messages: 4.080e+02 2.00000 3.060e+02 1.224e+03 >> MPI Message Lengths: 2.328e+06 2.00000 5.706e+03 6.984e+06 >> MPI Reductions: 8.750e+02 1.00000 >> >> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) >> e.g., VecAXPY() for real vectors of length N --> 2N flops >> and VecAXPY() for complex vectors of length N --> 8N flops >> >> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg %Total counts %Total Avg %Total counts %Total >> 0: Main Stage: 1.0644e+01 100.0% 1.8975e+09 100.0% 1.224e+03 100.0% 5.706e+03 100.0% 8.740e+02 99.9% >> >> ------------------------------------------------------------------------------------------------------------------------ >> See the 'Profiling' chapter of the users' manual for details on interpreting output. >> Phase summary info: >> Count: number of times phase was executed >> Time and Flops: Max - maximum over all processors >> Ratio - ratio of maximum to minimum over all processors >> Mess: number of messages sent >> Avg. len: average message length >> Reduct: number of global reductions >> Global: entire computation >> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). >> %T - percent time in this phase %f - percent flops in this phase >> %M - percent messages in this phase %L - percent message lengths in this phase >> %R - percent reductions in this phase >> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) >> ------------------------------------------------------------------------------------------------------------------------ >> Event Count Time (sec) Flops --- Global --- --- Stage --- Total >> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s >> ------------------------------------------------------------------------------------------------------------------------ >> --- Event Stage 0: Main Stage >> >> MatMult 202 1.0 5.5096e-01 1.0 1.38e+08 1.0 1.2e+03 5.7e+03 0.0e+00 5 29 99100 0 5 29 99100 0 998 >> MatSolve 252 1.0 6.9136e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 6 36 0 0 0 6 36 0 0 0 986 >> MatLUFactorNum 50 1.0 4.6002e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00 4 15 0 0 0 4 15 0 0 0 634 >> MatILUFactorSym 1 1.0 9.5899e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 50 1.0 1.6270e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 11 0 0 0 0 11 0 >> MatAssemblyEnd 50 1.0 1.0896e-01 1.0 0.00e+00 0.0 1.2e+01 1.4e+03 8.0e+00 1 0 1 0 1 1 0 1 0 1 0 >> MatGetRowIJ 1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatGetOrdering 1 1.0 7.2002e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> KSPSetUp 100 1.0 2.9130e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> KSPSolve 50 1.0 2.0737e+00 1.0 4.76e+08 1.0 1.2e+03 5.7e+03 4.6e+02 19100 99100 52 19100 99100 53 915 >> VecDot 202 1.0 7.3588e-02 1.1 1.63e+07 1.0 0.0e+00 0.0e+00 2.0e+02 1 3 0 0 23 1 3 0 0 23 885 >> VecDotNorm2 101 1.0 3.9155e-02 1.7 1.63e+07 1.0 0.0e+00 0.0e+00 1.0e+02 0 3 0 0 12 0 3 0 0 12 1664 >> VecNorm 151 1.0 5.8769e-02 1.7 1.22e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 3 0 0 17 0 3 0 0 17 829 >> VecCopy 100 1.0 2.3459e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecSet 403 1.0 5.9994e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >> VecAXPBYCZ 202 1.0 6.6376e-02 1.0 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 7 0 0 0 1 7 0 0 0 1963 >> VecWAXPY 202 1.0 6.9311e-02 1.0 1.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 940 >> VecAssemblyBegin 100 1.0 4.0355e-0214.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+02 0 0 0 0 34 0 0 0 0 34 0 >> VecAssemblyEnd 100 1.0 5.0378e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecScatterBegin 202 1.0 6.2275e-03 1.5 0.00e+00 0.0 1.2e+03 5.7e+03 0.0e+00 0 0 99100 0 0 0 99100 0 0 >> VecScatterEnd 202 1.0 2.0878e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> PCSetUp 100 1.0 4.7225e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 5.0e+00 4 15 0 0 1 4 15 0 0 1 617 >> PCSetUpOnBlocks 50 1.0 4.7191e-01 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 3.0e+00 4 15 0 0 0 4 15 0 0 0 618 >> PCApply 252 1.0 7.3425e-01 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 7 36 0 0 0 7 36 0 0 0 928 >> ------------------------------------------------------------------------------------------------------------------------ >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' Mem. >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> >> Matrix 4 4 16900896 0 >> Krylov Solver 2 2 2168 0 >> Vector 12 12 2604080 0 >> Vector Scatter 1 1 1060 0 >> Index Set 5 5 167904 0 >> Preconditioner 2 2 1800 0 >> Viewer 1 0 0 0 >> ======================================================================================================================== >> Average time to get PetscTime(): 1.09673e-06 >> Average time for MPI_Barrier(): 4.00543e-06 >> Average time for zero size MPI_Send(): 1.22786e-05 >> #PETSc Option Table entries: >> >> -log_summary >> -mg_ksp_view >> #End of PETSc Option Table entries >> Compiled without FORTRAN kernels >> Compiled with full precision matrices (default) >> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >> Configure run at: Thu May 31 09:53:43 2012 >> Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries >> ----------------------------------------- >> Libraries compiled on Thu May 31 09:53:43 2012 on hpc12 >> Machine characteristics: Linux-2.6.32-220.2.1.el6.x86_64-x86_64-with-centos-6.2-Final >> Using PETSc directory: /home/wtay/Codes/petsc-dev >> Using PETSc arch: petsc-3.2-dev_shared_rel >> ----------------------------------------- >> >> Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -Qoption,cpp,--extended_float_type -O3 ${COPTFLAGS} ${CFLAGS} >> Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} >> ----------------------------------------- >> >> Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/include -I/opt/openmpi-1.5.3/include >> ----------------------------------------- >> >> Using C linker: /opt/openmpi-1.5.3/bin/mpicc >> Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 >> Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_rel/lib -lHYPRE -lmpi_cxx -Wl,-rpath,/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lstdc++ -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl -L/opt/openmpi-1.5.3/lib -lmpi -lnsl -lutil -L/opt/intelcpro-11.1.059/lib/intel64 -limf -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl >> ----------------------------------------- >> >> Yours sincerely, >> >> TAY wee-beng >> >> >> On 5/6/2012 1:34 AM, Barry Smith wrote: >>> Also run with -ksp_view to see exasctly what solver options it is using. For example the number of levels, smoother on each level etc. My guess is that the below is running on one level (because I don't see you supplying options to control the number of levels etc). >>> >>> Barry >>> >>> On Jun 4, 2012, at 4:15 PM, Jed Brown wrote: >>> >>>> Always send -log_summary when asking about performance. >>>> >>>> On Mon, Jun 4, 2012 at 4:11 PM, TAY wee-beng wrote: >>>> Hi, >>>> >>>> I tried using PETSc multigrid on my 2D CFD code. I had converted ksp eg. ex29 to Fortran and then added into my code to solve the Poisson equation. >>>> >>>> The main subroutines are: >>>> >>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>> >>>> call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) >>>> call DMSetFunction(da,ComputeRHS,ierr) >>>> call DMSetJacobian(da,ComputeMatrix,ierr) >>>> call KSPSetDM(ksp,da,ierr) >>>> >>>> call KSPSetFromOptions(ksp,ierr) >>>> call KSPSetUp(ksp,ierr) >>>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr) >>>> call KSPGetSolution(ksp,x,ierr) >>>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr) >>>> call KSPDestroy(ksp,ierr) >>>> call DMDestroy(da,ierr) >>>> call PetscFinalize(ierr) >>>> >>>> >>>> Since the LHS matrix doesn't change, I only set up at the 1st time step, thereafter I only called ComputeRHS every time step. >>>> >>>> I was using HYPRE's geometric multigrid and the speed was much faster. >>>> >>>> What other options can I tweak to improve the speed? Or should I call the subroutines above at every timestep? >>>> >>>> Thanks! >>>> >>>> >>>> -- >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 6 21:33:49 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 6 Jun 2012 21:33:49 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FCFC9E4.3050301@gmail.com> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> Message-ID: On Wed, Jun 6, 2012 at 4:21 PM, TAY wee-beng wrote: > *call PCMGSetLevels(pc,mg_lvl,MPI_COMM_WORLD,ierr)* The third arguments is an array of length mg_lvl, not a single communicator. You can pass PETSC_NULL_OBJECT just like the man page says to use the default. > > > However, I get the error: > > Caught signal number 11 SEGV: Segmentation Violation, probably memory > access out of range > > after calling *PCMGSetLevels* > > What's the problem? Is there any examples which I can follow? > I believe the other examples that use this routine are in C or just tests (not tutorial-style) examples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Jun 7 02:35:01 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 07 Jun 2012 09:35:01 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> Message-ID: <4FD059A5.3030300@gmail.com> On 7/6/2012 4:33 AM, Jed Brown wrote: > On Wed, Jun 6, 2012 at 4:21 PM, TAY wee-beng > wrote: > > *call PCMGSetLevels(pc,mg_lvl,MPI_COMM_WORLD,ierr)* > > > The third arguments is an array of length mg_lvl, not a single > communicator. You can pass PETSC_NULL_OBJECT just like the man page > says to use the default. > > I changed but the same Segmentation still occurs: call KSPCreate(MPI_COMM_WORLD,ksp,ierr) call KSPGetPC(ksp,pc,ierr) call PCSetType(pc_uv,PCMG,ierr) mg_lvl = 1 (or 2) call PCMGSetLevels(pc,mg_lvl,PETSC_NULL_OBJECT,ierr) call DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,1,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) ... Btw, I tried to look at http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex42.c.html but I think there's some error in the page formatting. > > > However, I get the error: > > Caught signal number 11 SEGV: Segmentation Violation, probably > memory access out of range > > after calling *PCMGSetLevels* > > What's the problem? Is there any examples which I can follow? > > > I believe the other examples that use this routine are in C or just > tests (not tutorial-style) examples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 7 06:20:56 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 7 Jun 2012 19:20:56 +0800 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FD059A5.3030300@gmail.com> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> <4FD059A5.3030300@gmail.com> Message-ID: On Thu, Jun 7, 2012 at 3:35 PM, TAY wee-beng wrote: > > On 7/6/2012 4:33 AM, Jed Brown wrote: > > On Wed, Jun 6, 2012 at 4:21 PM, TAY wee-beng wrote: > >> *call PCMGSetLevels(pc,mg_lvl,MPI_COMM_WORLD,ierr)* > > > The third arguments is an array of length mg_lvl, not a single > communicator. You can pass PETSC_NULL_OBJECT just like the man page says to > use the default. > > >> >> I changed but the same Segmentation still occurs: > Look, programming is a skill. It demands you learn to use certain tools. A message like "Segmentation still occurs" is USELESS since we are not looking at your code or running it. Sending in a stack trace from gdb is much more informative and means you will get help sooner. Matt > call KSPCreate(MPI_COMM_WORLD,ksp,ierr) > > call KSPGetPC(ksp,pc,ierr) > > call PCSetType(pc_uv,PCMG,ierr) > > mg_lvl = 1 (or 2) > > call PCMGSetLevels(pc,mg_lvl,PETSC_NULL_OBJECT,ierr) > > call > DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,1,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > > ... > > Btw, I tried to look at > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex42.c.htmlbut I think there's some error in the page formatting. > > >> However, I get the error: >> >> Caught signal number 11 SEGV: Segmentation Violation, probably memory >> access out of range >> >> after calling *PCMGSetLevels* >> >> What's the problem? Is there any examples which I can follow? >> > > I believe the other examples that use this routine are in C or just tests > (not tutorial-style) examples. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 7 06:41:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 7 Jun 2012 06:41:29 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FD059A5.3030300@gmail.com> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> <4FD059A5.3030300@gmail.com> Message-ID: On Thu, Jun 7, 2012 at 2:35 AM, TAY wee-beng wrote: > Btw, I tried to look at > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex42.c.htmlbut I think there's some error in the page formatting. > I fixed this, but it didn't propagate to the web site yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Jun 7 16:56:59 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 07 Jun 2012 23:56:59 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> <4FD059A5.3030300@gmail.com> Message-ID: <4FD123AB.1070009@gmail.com> On 7/6/2012 1:20 PM, Matthew Knepley wrote: > On Thu, Jun 7, 2012 at 3:35 PM, TAY wee-beng > wrote: > > > On 7/6/2012 4:33 AM, Jed Brown wrote: >> On Wed, Jun 6, 2012 at 4:21 PM, TAY wee-beng > > wrote: >> >> *call PCMGSetLevels(pc,mg_lvl,MPI_COMM_WORLD,ierr)* >> >> >> The third arguments is an array of length mg_lvl, not a single >> communicator. You can pass PETSC_NULL_OBJECT just like the man >> page says to use the default. >> >> > I changed but the same Segmentation still occurs: > > > Look, programming is a skill. It demands you learn to use certain > tools. A message like "Segmentation still occurs" > is USELESS since we are not looking at your code or running it. > Sending in a stack trace from gdb is much more > informative and means you will get help sooner. > > Matt I have tried to troubleshoot and found the problem. Now after adding *PCMGSetLevels* with mg_lvl = 1 and using *-log_summary -mg_ksp_view* ( with call KSPSetOptionsPrefix(ksp,"mg_",ierr)), I got the output below. I looked at the manual but I'm not sure how to get better performance. Also, what are the more common options to start with. Is there an appropriate C example? Some options are: *PCMGSetLevels* - how many lvls are appropriate? PCMGSetCycleType - PCMGSetNumberSmoothUp/down etc ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.2 named n12-58 with 4 processors, by wtay Thu Jun 7 23:00:37 2012 Using Petsc Development HG revision: c76fb3cac2a4ad0dfc9436df80f678898c867e86 HG Date: Thu May 31 00:33:26 2012 -0500 Max Max/Min Avg Total Time (sec): 8.522e+01 1.00001 8.522e+01 Objects: 2.700e+01 1.00000 2.700e+01 Flops: 4.756e+08 1.00811 4.744e+08 1.897e+09 Flops/sec: 5.580e+06 1.00812 5.566e+06 2.227e+07 Memory: 2.075e+07 1.00333 8.291e+07 MPI Messages: 4.080e+02 2.00000 3.060e+02 1.224e+03 MPI Message Lengths: 2.328e+06 2.00000 5.706e+03 6.984e+06 MPI Reductions: 3.057e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 8.5219e+01 100.0% 1.8975e+09 100.0% 1.224e+03 100.0% 5.706e+03 100.0% 3.056e+03 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ ########################################################## # # # WARNING!!! # # # # This code was compiled with a debugging option, # # To get timing results run ./configure # # using --with-debugging=no, the performance will # # be generally two or three times faster. # # # ########################################################## Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 202 1.0 3.0738e+00 1.2 1.38e+08 1.0 1.2e+03 5.7e+03 0.0e+00 3 29 99100 0 3 29 99100 0 179 MatSolve 252 1.0 1.7658e+00 1.1 1.71e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 36 0 0 0 2 36 0 0 0 386 MatLUFactorNum 50 1.0 2.3908e+00 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 15 0 0 0 3 15 0 0 0 122 MatILUFactorSym 1 1.0 2.5288e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 50 1.0 1.6280e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 0 0 0 0 3 0 0 0 0 3 0 MatAssemblyEnd 50 1.0 4.1831e-01 1.0 0.00e+00 0.0 1.2e+01 1.4e+03 2.2e+02 0 0 1 0 7 0 0 1 0 7 0 MatGetRowIJ 1 1.0 4.0531e-06 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.6429e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 100 1.0 4.1475e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 50 1.0 1.8577e+01 1.0 4.76e+08 1.0 1.2e+03 5.7e+03 1.9e+03 22100 99100 63 22100 99100 63 102 VecDot 202 1.0 1.0362e+00 1.4 1.63e+07 1.0 0.0e+00 0.0e+00 2.0e+02 1 3 0 0 7 1 3 0 0 7 63 VecDotNorm2 101 1.0 1.7485e+00 2.6 1.63e+07 1.0 0.0e+00 0.0e+00 1.0e+02 1 3 0 0 3 1 3 0 0 3 37 VecNorm 151 1.0 1.6854e+00 1.1 1.22e+07 1.0 0.0e+00 0.0e+00 1.5e+02 2 3 0 0 5 2 3 0 0 5 29 VecCopy 100 1.0 7.1418e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 403 1.0 1.7004e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 202 1.0 3.0207e-01 1.5 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 7 0 0 0 0 7 0 0 0 431 VecWAXPY 202 1.0 3.2482e-01 1.4 1.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 201 VecAssemblyBegin 100 1.0 3.3056e+00 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+02 2 0 0 0 10 2 0 0 0 10 0 VecAssemblyEnd 100 1.0 9.0289e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 202 1.0 3.1948e-02 2.6 0.00e+00 0.0 1.2e+03 5.7e+03 0.0e+00 0 0 99100 0 0 0 99100 0 0 VecScatterEnd 202 1.0 9.4827e-01 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 PCSetUp 100 1.0 2.4949e+00 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 8.0e+00 3 15 0 0 0 3 15 0 0 0 117 PCSetUpOnBlocks 50 1.0 2.4723e+00 1.0 7.31e+07 1.0 0.0e+00 0.0e+00 4.0e+00 3 15 0 0 0 3 15 0 0 0 118 PCApply 252 1.0 3.7255e+00 1.4 1.71e+08 1.0 0.0e+00 0.0e+00 5.0e+02 4 36 0 0 16 4 36 0 0 16 183 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 4 4 16900896 0 Krylov Solver 2 2 2168 0 Vector 12 12 2604080 0 Vector Scatter 1 1 1060 0 Index Set 5 5 167904 0 Preconditioner 2 2 1800 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 5.57899e-06 Average time for zero size MPI_Send(): 2.37226e-05 #PETSc Option Table entries: -log_summary -mg_ksp_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Thu May 31 10:24:12 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Thu May 31 10:24:12 2012 on hpc12 Machine characteristics: Linux-2.6.32-220.2.1.el6.x86_64-x86_64-with-centos-6.2-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.2-dev_shared_debug ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -Qoption,cpp,--extended_float_type -g ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -g ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/lib -L/home/wtay/Codes/petsc-dev/petsc-3.2-dev_shared_debug/lib -lHYPRE -lmpi_cxx -Wl,-rpath,/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lstdc++ -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl -L/opt/openmpi-1.5.3/lib -lmpi -lnsl -lutil -L/opt/intelcpro-11.1.059/lib/intel64 -limf -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- > call KSPCreate(MPI_COMM_WORLD,ksp,ierr) > > call KSPGetPC(ksp,pc,ierr) > > call PCSetType(pc_uv,PCMG,ierr) > > mg_lvl = 1 (or 2) > > call PCMGSetLevels(pc,mg_lvl,PETSC_NULL_OBJECT,ierr) > > call > DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,1,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > > ... > > Btw, I tried to look at > http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex42.c.html > but I think there's some error in the page formatting. > >> >> However, I get the error: >> >> Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> >> after calling *PCMGSetLevels* >> >> What's the problem? Is there any examples which I can follow? >> >> >> I believe the other examples that use this routine are in C or >> just tests (not tutorial-style) examples. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 7 17:06:35 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 7 Jun 2012 17:06:35 -0500 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: <4FD123AB.1070009@gmail.com> References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> <4FD059A5.3030300@gmail.com> <4FD123AB.1070009@gmail.com> Message-ID: On Thu, Jun 7, 2012 at 4:56 PM, TAY wee-beng wrote: > I looked at the manual but I'm not sure how to get better performance. > > * ##########################################################* * # #* * # WARNING!!! #* * # #* * # This code was compiled with a debugging option, #* * # To get timing results run ./configure #* * # using --with-debugging=no, the performance will #* * # be generally two or three times faster. #* * # #* * ##########################################################* I wonder if making this text bigger, red, bold, and blinking (which I can't seem to do in this email) would increase the chances that you read it. > Also, what are the more common options to start with. Is there an appropriate C example? Some options are: > *PCMGSetLevels* - how many lvls are appropriate? > > PCMGSetCycleType - > > PCMGSetNumberSmoothUp/down etc > > 1. Control this stuff using command line options. Recompiling to run a different method was obsolete decades ago. 2. You need to understand a little about the math and a little about the performance of each component in isolation. I recommend seeing how time moves between different operations as you modify the algorithm. 3. Experiment with your machine and the methods. Learn which configurations have good algorithmic performance (few iterations) and which execute most efficiently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Thu Jun 7 23:26:16 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Thu, 7 Jun 2012 22:26:16 -0600 Subject: [petsc-users] segfault running src/dm/examples/tutorials/ex11f90.F In-Reply-To: References: Message-ID: I guess that the petsc Fortran 90 feature doesn't work with intel fortran compiler. Is that correct? Thanks, Panruo Wu On Tue, Jun 5, 2012 at 9:54 PM, Panruo Wu wrote: > Dear list, > > I run into a segfault when running src/dm/examples/tutorials/ex11f90.F > > Intel Fortran Compiler 12.1.3; 1.6 OpenMPI; Ubuntu 12.04 > other examples compiles and runs well. > > It compiles without any warnings or errors; however it cannot run. > Any ideas? Error message are attached. > > Thank you! > Panruo Wu > > $ mpirun -np 1 ex11f90 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ex11f90 on a opmi_inte named avahome by pwu Tue Jun 5 > 21:44:11 2012 > [0]PETSC ERROR: Libraries linked from > /home/pwu/Downloads/petsc-3.2-p7/opmi_intel_dbg/lib > [0]PETSC ERROR: Configure run at Tue Jun 5 10:02:00 2012 > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64 --download-umfpack > --download-superlu --with-x=1 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > mpirun has exited due to process rank 0 with PID 22293 on > node avahome exiting improperly. There are two reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > This may have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 7 23:45:15 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 7 Jun 2012 23:45:15 -0500 Subject: [petsc-users] segfault running src/dm/examples/tutorials/ex11f90.F In-Reply-To: References: Message-ID: <6C169B3C-5C2F-43D1-9E4E-861E47572244@mcs.anl.gov> Please switch to petsc-3.3 (just released) http://www.mcs.anl.gov/petsc/download/index.html If the problem persists then send configure.log make.log and all the output from running the example (with petsc-3.3) to petsc-maint at mcs.anl.gov Barry On Jun 5, 2012, at 10:54 PM, Panruo Wu wrote: > Dear list, > > I run into a segfault when running src/dm/examples/tutorials/ex11f90.F > > Intel Fortran Compiler 12.1.3; 1.6 OpenMPI; Ubuntu 12.04 > other examples compiles and runs well. > > It compiles without any warnings or errors; however it cannot run. > Any ideas? Error message are attached. > > Thank you! > Panruo Wu > > $ mpirun -np 1 ex11f90 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ex11f90 on a opmi_inte named avahome by pwu Tue Jun 5 21:44:11 2012 > [0]PETSC ERROR: Libraries linked from /home/pwu/Downloads/petsc-3.2-p7/opmi_intel_dbg/lib > [0]PETSC ERROR: Configure run at Tue Jun 5 10:02:00 2012 > [0]PETSC ERROR: Configure options --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64 --download-umfpack --download-superlu --with-x=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > -------------------------------------------------------------------------- > mpirun has exited due to process rank 0 with PID 22293 on > node avahome exiting improperly. There are two reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > This may have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > -------------------------------------------------------------------------- > From w_ang_temp at 163.com Fri Jun 8 01:31:21 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Fri, 8 Jun 2012 14:31:21 +0800 (CST) Subject: [petsc-users] From CSR format to MatSetValues Message-ID: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> Hello, When solving a linear system Ax=b, the first step is assigning values to matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove Ax=b with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set values to PETSc Mat A. The variables 'values'?'columns'?'rowIndex' belong to the main function. They are used to represent a matrix in CSR format. The following table describes the arrays in terms of the values, row, and column positions of the non-zero elements in a sparse matrix. values: A real or complex array that contains the non-zero elements of a sparse matrix. The non-zero elements are mapped into the values array using the row-major upper triangular storage mapping described above. columns: Element i of the integer array columns is the number of the column that contains the i-th element in the values array. rowIndex: Element j of the integer array rowIndex gives the index of the element in the values array that is first non-zero element in a row j. codes: ---------Set Values to A From CSR Format(values,rowIndex,columns)------------------- !values:Non-Symmetric Matrix ione = 1 do 10,II=Istart+1,Iend !non-zero numbers of this row rowNum=rowIndex(II+1)-rowIndex(II) do 11,JJ=1,rowNum !elemnt index of the values/columns kValNn=rowIndex(II)+JJ-1 !column index of this element nCol=columns(kValNn)-1 !Setdata:II-RowIndex,nCol-ColIndex nRow=II-1 call MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) 11 continue 10 continue -------------------------------------------------------------------------------------- As we can see, it has to loop a number of times because it can only set one data per time. And this leads to low efficiency. So is there a better way to do this? Thank you. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 8 07:20:22 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 8 Jun 2012 20:20:22 +0800 Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> Message-ID: On Fri, Jun 8, 2012 at 2:31 PM, w_ang_temp wrote: > Hello, > When solving a linear system Ax=b, the first step is assigning values > to > matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove > Ax=b > with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set > values > to PETSc Mat A. > The variables 'values'?'columns'?'rowIndex' belong to the main > function. > They are used to represent a matrix in CSR format. The following table > describes > the arrays in terms of the values, row, and column positions of the > non-zero > elements in a sparse matrix. > values: A real or complex array that contains the non-zero elements of > a > sparse matrix. The non-zero elements are mapped into the values > array using the row-major upper triangular storage mapping > described > above. > columns: Element i of the integer array columns is the number of the > column > that contains the i-th element in the values array. > rowIndex: Element j of the integer array rowIndex gives the index of > the > element in the values array that is first non-zero element > in a row j. > > codes: > ---------Set Values to A From CSR > Format(values,rowIndex,columns)------------------- > !values:Non-Symmetric Matrix > ione = 1 > do 10,II=Istart+1,Iend > !non-zero numbers of this row > rowNum=rowIndex(II+1)-rowIndex(II) > do 11,JJ=1,rowNum > > !elemnt index of the values/columns > kValNn=rowIndex(II)+JJ-1 > !column index of this element > nCol=columns(kValNn)-1 > > !Setdata:II-RowIndex,nCol-ColIndex > nRow=II-1 > call > MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) > 11 continue > 10 continue > > -------------------------------------------------------------------------------------- > > As we can see, it has to loop a number of times because it can only > set one data per > time. And this leads to low efficiency. > Almost certainly, this is not your problem. If anything is slow, its a problem with preallocation. > So is there a better way to do this? > You can go directly to the data structure in serial, but then you give up on parallelism. Matt > Thank you. > Jim > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Fri Jun 8 07:38:13 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Fri, 08 Jun 2012 07:38:13 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] Message-ID: <4FD1F235.9080800@geology.wisc.edu> Sorry about a question not directly related to PETSc but has anyone here been able to use the METIS 5.0 (that PETSc 3.3/dev downloads/builds) with Fortran? There has been an API change from 4 to 5 but I am having some trouble and METIS manual/forums havent been useful. For example consider the simple code (below) that partitions a two element mesh made of linear quads into two. The elements are numbered 0 1 2 3 and 1 4 5 2.It works fine with GNU FC (no valgrind errors). With Intel FC it works fines (though valgrind throws a bunch of errors). However with PGI compilers I get a segfault. program test implicit none integer, parameter :: nels=2, nnds=6, npel=4 integer :: eptr(nels+1), nodes(nels*npel), epart(nels), npart(nnds), n integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), mopts(:)=>null() real(8), pointer :: tpwgts(:)=>null() eptr=(/0,4,7/) nodes=(/0,1,2,3,1,4,5,2/) call METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) print*, npart; print*, epart end program test According to the manual moving from METIS 4 to 5 only involves passing some additional nulls. I am not sure what I missed. http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf Thanks in advance. Tabrez From john.fettig at gmail.com Fri Jun 8 08:34:28 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 8 Jun 2012 09:34:28 -0400 Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> Message-ID: On Fri, Jun 8, 2012 at 2:31 AM, w_ang_temp wrote: > Hello, > ??? When solving a linear system Ax=b, the first step is assigning values to > matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove > Ax=b > with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set > values > to PETSc Mat A. > ??? The variables 'values'?'columns'?'rowIndex' belong to the main function. > They are used to represent a matrix in CSR format. The following table > describes > the arrays in terms of the values, row, and column positions of the non-zero > elements in a sparse matrix. > ??? values: A real or complex array that contains the non-zero elements of a > ??????????? sparse matrix. The non-zero elements are mapped into the values > ??????????? array using the row-major upper triangular storage mapping > described > ??????????? above. > ??? columns: Element i of the integer array columns is the number of the > column > ???????????? that contains the i-th element in the values array. > ??? rowIndex: Element j of the integer array rowIndex gives the index of the > ????????????? element in the values array that is first non-zero element in > a row j. > > codes: > ---------Set Values to A From CSR > Format(values,rowIndex,columns)------------------- > ????? !values:Non-Symmetric Matrix > ????? ione = 1 > ????? do 10,II=Istart+1,Iend > ??????? !non-zero numbers of this row > ??????? rowNum=rowIndex(II+1)-rowIndex(II) > ??????? do 11,JJ=1,rowNum > > ??????????? !elemnt index of the values/columns > ??????????? kValNn=rowIndex(II)+JJ-1 > ??????????? !column index of this element > ??????????? nCol=columns(kValNn)-1 > > ??????????? !Setdata:II-RowIndex,nCol-ColIndex > ??????????? nRow=II-1 > ??????????? call > MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) > ? 11??? continue > ? 10? continue > -------------------------------------------------------------------------------------- > > ??? As we can see, it has to loop a number of times because it can only set > one data per > time. And this leads to low efficiency. Why not call MatSetValues once per row? John From john.mousel at gmail.com Fri Jun 8 08:39:27 2012 From: john.mousel at gmail.com (John Mousel) Date: Fri, 8 Jun 2012 08:39:27 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: <4FD1F235.9080800@geology.wisc.edu> References: <4FD1F235.9080800@geology.wisc.edu> Message-ID: It's hard to tell from the info you provided, but you seem to be playing fast and loose with your type declarations. METIS is expecting real_t, which is a 32 bit real if you haven't changed the definition in metis.h. I know this has caused me problems in the past. On Fri, Jun 8, 2012 at 7:38 AM, Tabrez Ali wrote: > Sorry about a question not directly related to PETSc but has anyone here > been able to use the METIS 5.0 (that PETSc 3.3/dev downloads/builds) with > Fortran? There has been an API change from 4 to 5 but I am having some > trouble and METIS manual/forums havent been useful. > > For example consider the simple code (below) that partitions a two element > mesh made of linear quads into two. The elements are numbered 0 1 2 3 and 1 > 4 5 2.It works fine with GNU FC (no valgrind errors). With Intel FC it > works fines (though valgrind throws a bunch of errors). However with PGI > compilers I get a segfault. > > program test > implicit none > integer, parameter :: nels=2, nnds=6, npel=4 > integer :: eptr(nels+1), nodes(nels*npel), epart(nels), > npart(nnds), n > integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), > mopts(:)=>null() > real(8), pointer :: tpwgts(:)=>null() > eptr=(/0,4,7/) > nodes=(/0,1,2,3,1,4,5,2/) > call METIS_PartMeshNodal(nels,nnds,**eptr,nodes,vwgt,vsize,2,** > tpwgts,mopts,n,epart,npart) > print*, npart; print*, epart > end program test > > According to the manual moving from METIS 4 to 5 only involves passing > some additional nulls. I am not sure what I missed. > > http://glaros.dtc.umn.edu/**gkhome/fetch/sw/metis/manual.**pdf > > Thanks in advance. > > Tabrez > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Fri Jun 8 09:02:33 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Fri, 08 Jun 2012 09:02:33 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: References: <4FD1F235.9080800@geology.wisc.edu> Message-ID: <4FD205F9.2030807@geology.wisc.edu> Thanks for your answer. Unfortunately its not that. Based on past experience I am sure I am overlooking something very simple but I cant seem to find out what. Btw METIS 4 worked fine before. On 06/08/2012 08:39 AM, John Mousel wrote: > It's hard to tell from the info you provided, but you seem to be > playing fast and loose with your type declarations. METIS is expecting > real_t, which is a 32 bit real if you haven't changed the definition > in metis.h. I know this has caused me problems in the past. > > On Fri, Jun 8, 2012 at 7:38 AM, Tabrez Ali > wrote: > > Sorry about a question not directly related to PETSc but has > anyone here been able to use the METIS 5.0 (that PETSc 3.3/dev > downloads/builds) with Fortran? There has been an API change from > 4 to 5 but I am having some trouble and METIS manual/forums havent > been useful. > > For example consider the simple code (below) that partitions a two > element mesh made of linear quads into two. The elements are > numbered 0 1 2 3 and 1 4 5 2.It works fine with GNU FC (no > valgrind errors). With Intel FC it works fines (though valgrind > throws a bunch of errors). However with PGI compilers I get a > segfault. > > program test > implicit none > integer, parameter :: nels=2, nnds=6, npel=4 > integer :: eptr(nels+1), nodes(nels*npel), > epart(nels), npart(nnds), n > integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), > mopts(:)=>null() > real(8), pointer :: tpwgts(:)=>null() > eptr=(/0,4,7/) > nodes=(/0,1,2,3,1,4,5,2/) > call > METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) > print*, npart; print*, epart > end program test > > According to the manual moving from METIS 4 to 5 only involves > passing some additional nulls. I am not sure what I missed. > > http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf > > Thanks in advance. > > Tabrez > -- No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.mousel at gmail.com Fri Jun 8 09:15:53 2012 From: john.mousel at gmail.com (John Mousel) Date: Fri, 8 Jun 2012 09:15:53 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: <4FD205F9.2030807@geology.wisc.edu> References: <4FD1F235.9080800@geology.wisc.edu> <4FD205F9.2030807@geology.wisc.edu> Message-ID: Attaching the Valgrind output would probably help a lot. On Fri, Jun 8, 2012 at 9:02 AM, Tabrez Ali wrote: > Thanks for your answer. Unfortunately its not that. > > Based on past experience I am sure I am overlooking something very simple > but I cant seem to find out what. Btw METIS 4 worked fine before. > > On 06/08/2012 08:39 AM, John Mousel wrote: > > It's hard to tell from the info you provided, but you seem to be playing > fast and loose with your type declarations. METIS is expecting real_t, > which is a 32 bit real if you haven't changed the definition in metis.h. I > know this has caused me problems in the past. > > On Fri, Jun 8, 2012 at 7:38 AM, Tabrez Ali wrote: > >> Sorry about a question not directly related to PETSc but has anyone here >> been able to use the METIS 5.0 (that PETSc 3.3/dev downloads/builds) with >> Fortran? There has been an API change from 4 to 5 but I am having some >> trouble and METIS manual/forums havent been useful. >> >> For example consider the simple code (below) that partitions a two >> element mesh made of linear quads into two. The elements are numbered 0 1 2 >> 3 and 1 4 5 2.It works fine with GNU FC (no valgrind errors). With Intel FC >> it works fines (though valgrind throws a bunch of errors). However with PGI >> compilers I get a segfault. >> >> program test >> implicit none >> integer, parameter :: nels=2, nnds=6, npel=4 >> integer :: eptr(nels+1), nodes(nels*npel), epart(nels), >> npart(nnds), n >> integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), >> mopts(:)=>null() >> real(8), pointer :: tpwgts(:)=>null() >> eptr=(/0,4,7/) >> nodes=(/0,1,2,3,1,4,5,2/) >> call >> METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) >> print*, npart; print*, epart >> end program test >> >> According to the manual moving from METIS 4 to 5 only involves passing >> some additional nulls. I am not sure what I missed. >> >> http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf >> >> Thanks in advance. >> >> Tabrez >> > > -- > No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwu at mymail.mines.edu Fri Jun 8 09:43:52 2012 From: pwu at mymail.mines.edu (Panruo Wu) Date: Fri, 8 Jun 2012 08:43:52 -0600 Subject: [petsc-users] segfault running src/dm/examples/tutorials/ex11f90.F In-Reply-To: <6C169B3C-5C2F-43D1-9E4E-861E47572244@mcs.anl.gov> References: <6C169B3C-5C2F-43D1-9E4E-861E47572244@mcs.anl.gov> Message-ID: The problem still persists with petsc-3.3. I've send the bug report to peetsc-maint. Thanks & regards Panruo Wu On Thu, Jun 7, 2012 at 10:45 PM, Barry Smith wrote: > > Please switch to petsc-3.3 (just released) > http://www.mcs.anl.gov/petsc/download/index.html > > If the problem persists then send configure.log make.log and all the > output from running the example (with petsc-3.3) to > petsc-maint at mcs.anl.gov > > Barry > > On Jun 5, 2012, at 10:54 PM, Panruo Wu wrote: > > > Dear list, > > > > I run into a segfault when running src/dm/examples/tutorials/ex11f90.F > > > > Intel Fortran Compiler 12.1.3; 1.6 OpenMPI; Ubuntu 12.04 > > other examples compiles and runs well. > > > > It compiles without any warnings or errors; however it cannot run. > > Any ideas? Error message are attached. > > > > Thank you! > > Panruo Wu > > > > $ mpirun -np 1 ex11f90 > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 > 09:30:51 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ex11f90 on a opmi_inte named avahome by pwu Tue Jun 5 > 21:44:11 2012 > > [0]PETSC ERROR: Libraries linked from > /home/pwu/Downloads/petsc-3.2-p7/opmi_intel_dbg/lib > > [0]PETSC ERROR: Configure run at Tue Jun 5 10:02:00 2012 > > [0]PETSC ERROR: Configure options > --with-blas-lapack-dir=/opt/intel/mkl/lib/intel64 --download-umfpack > --download-superlu --with-x=1 > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 59. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > > -------------------------------------------------------------------------- > > > -------------------------------------------------------------------------- > > mpirun has exited due to process rank 0 with PID 22293 on > > node avahome exiting improperly. There are two reasons this could occur: > > > > 1. this process did not call "init" before exiting, but others in > > the job did. This can cause a job to hang indefinitely while it waits > > for all processes to call "init". By rule, if one process calls "init", > > then ALL processes must call "init" prior to termination. > > > > 2. this process called "init", but exited without calling "finalize". > > By rule, all processes that call "init" MUST call "finalize" prior to > > exiting or it will be considered an "abnormal termination" > > > > This may have caused other processes in the application to be > > terminated by signals sent by mpirun (as reported here). > > > -------------------------------------------------------------------------- > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Fri Jun 8 10:35:45 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Fri, 08 Jun 2012 10:35:45 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: References: <4FD1F235.9080800@geology.wisc.edu> <4FD205F9.2030807@geology.wisc.edu> Message-ID: <4FD21BD1.9020401@geology.wisc.edu> Here it is for the 3 compilers. GNU FC http://bpaste.net/show/31029/ Intel FC http://bpaste.net/show/QJ6k5jc52t7MGUlvrlEV/ PGI FC http://bpaste.net/show/DZzY6Re5iXMgLSHydOgJ/ On 06/08/2012 09:15 AM, John Mousel wrote: > Attaching the Valgrind output would probably help a lot. > > On Fri, Jun 8, 2012 at 9:02 AM, Tabrez Ali > wrote: > > Thanks for your answer. Unfortunately its not that. > > Based on past experience I am sure I am overlooking something very > simple but I cant seem to find out what. Btw METIS 4 worked fine > before. > > On 06/08/2012 08:39 AM, John Mousel wrote: >> It's hard to tell from the info you provided, but you seem to be >> playing fast and loose with your type declarations. METIS is >> expecting real_t, which is a 32 bit real if you haven't changed >> the definition in metis.h. I know this has caused me problems in >> the past. >> >> On Fri, Jun 8, 2012 at 7:38 AM, Tabrez Ali >> > wrote: >> >> Sorry about a question not directly related to PETSc but has >> anyone here been able to use the METIS 5.0 (that PETSc >> 3.3/dev downloads/builds) with Fortran? There has been an API >> change from 4 to 5 but I am having some trouble and METIS >> manual/forums havent been useful. >> >> For example consider the simple code (below) that partitions >> a two element mesh made of linear quads into two. The >> elements are numbered 0 1 2 3 and 1 4 5 2.It works fine with >> GNU FC (no valgrind errors). With Intel FC it works fines >> (though valgrind throws a bunch of errors). However with PGI >> compilers I get a segfault. >> >> program test >> implicit none >> integer, parameter :: nels=2, nnds=6, npel=4 >> integer :: eptr(nels+1), nodes(nels*npel), >> epart(nels), npart(nnds), n >> integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), >> mopts(:)=>null() >> real(8), pointer :: tpwgts(:)=>null() >> eptr=(/0,4,7/) >> nodes=(/0,1,2,3,1,4,5,2/) >> call >> METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) >> print*, npart; print*, epart >> end program test >> >> According to the manual moving from METIS 4 to 5 only >> involves passing some additional nulls. I am not sure what I >> missed. >> >> http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf >> >> Thanks in advance. >> >> Tabrez >> > > -- > No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s_g at berkeley.edu Fri Jun 8 10:30:18 2012 From: s_g at berkeley.edu (Sanjay Govindjee) Date: Fri, 08 Jun 2012 08:30:18 -0700 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: <4FD21BD1.9020401@geology.wisc.edu> References: <4FD1F235.9080800@geology.wisc.edu> <4FD205F9.2030807@geology.wisc.edu> <4FD21BD1.9020401@geology.wisc.edu> Message-ID: <4FD21A8A.60206@berkeley.edu> I tried to make Metis 5 work with fortran for quite some time but was not able to make it work; in the end there were some things that I could not make work so I ended up with a C function which I call out of fortran; the C function is attached -- calling it is straightforward just pass in the indicated integers and arrays. [I went this route, because in the end I think I figured out that the Fortran interface in Metis 5 was missing some feature to allow it work properly -- the details escape me at the moment since I did this about 1 month back.] -sanjay On 6/8/12 8:35 AM, Tabrez Ali wrote: > Here it is for the 3 compilers. > > GNU FC http://bpaste.net/show/31029/ > > Intel FC http://bpaste.net/show/QJ6k5jc52t7MGUlvrlEV/ > > PGI FC http://bpaste.net/show/DZzY6Re5iXMgLSHydOgJ/ > > > On 06/08/2012 09:15 AM, John Mousel wrote: >> Attaching the Valgrind output would probably help a lot. >> >> On Fri, Jun 8, 2012 at 9:02 AM, Tabrez Ali > > wrote: >> >> Thanks for your answer. Unfortunately its not that. >> >> Based on past experience I am sure I am overlooking something >> very simple but I cant seem to find out what. Btw METIS 4 worked >> fine before. >> >> On 06/08/2012 08:39 AM, John Mousel wrote: >>> It's hard to tell from the info you provided, but you seem to be >>> playing fast and loose with your type declarations. METIS is >>> expecting real_t, which is a 32 bit real if you haven't changed >>> the definition in metis.h. I know this has caused me problems in >>> the past. >>> >>> On Fri, Jun 8, 2012 at 7:38 AM, Tabrez Ali >>> > wrote: >>> >>> Sorry about a question not directly related to PETSc but has >>> anyone here been able to use the METIS 5.0 (that PETSc >>> 3.3/dev downloads/builds) with Fortran? There has been an >>> API change from 4 to 5 but I am having some trouble and >>> METIS manual/forums havent been useful. >>> >>> For example consider the simple code (below) that partitions >>> a two element mesh made of linear quads into two. The >>> elements are numbered 0 1 2 3 and 1 4 5 2.It works fine with >>> GNU FC (no valgrind errors). With Intel FC it works fines >>> (though valgrind throws a bunch of errors). However with PGI >>> compilers I get a segfault. >>> >>> program test >>> implicit none >>> integer, parameter :: nels=2, nnds=6, npel=4 >>> integer :: eptr(nels+1), nodes(nels*npel), >>> epart(nels), npart(nnds), n >>> integer, pointer :: vwgt(:)=>null(), vsize(:)=>null(), >>> mopts(:)=>null() >>> real(8), pointer :: tpwgts(:)=>null() >>> eptr=(/0,4,7/) >>> nodes=(/0,1,2,3,1,4,5,2/) >>> call >>> METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) >>> print*, npart; print*, epart >>> end program test >>> >>> According to the manual moving from METIS 4 to 5 only >>> involves passing some additional nulls. I am not sure what I >>> missed. >>> >>> http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf >>> >>> Thanks in advance. >>> >>> Tabrez >>> >> >> -- >> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- /* c * * F E A P * * A Finite Element Analysis Program c.... Copyright (c) 1984-2012: Regents of the University of California c All rights reserved c-----[--.----+----.----+----.-----------------------------------------] c Modification log Date (dd/mm/year) c Original version 06/05/2012 c-----[--.----+----.----+----.-----------------------------------------] c Purpose: C cover function to partition mesh using METIS ver 5.x c Inputs: c numnp -- number of nodes c xadj -- pointers into adjaceny matrix c adjncy -- adjaceny matrix for node graph c domains -- number of domains to partition graph into c Outputs: c part -- partitioned graph c-----[--.----+----.----+----.-----------------------------------------] */ #include #include "metis.h" int smetis_(int *numnp,int *xadj,int *adjncy,int *domains, int *part) { int rval; int ncon = 1; int edgecut = 0; int options[METIS_NOPTIONS]; // Default options METIS_SetDefaultOptions(options); options[METIS_OPTION_OBJTYPE] = 0; options[METIS_OPTION_CTYPE] = 1; options[METIS_OPTION_DBGLVL] = 0; options[METIS_OPTION_NITER] = 10; options[METIS_OPTION_NCUTS] = 1; options[METIS_OPTION_MINCONN] = 0; options[METIS_OPTION_CONTIG] = 0; options[METIS_OPTION_NUMBERING] = 1; if (*domains < 8) { // Recursive options[METIS_OPTION_IPTYPE] = 0; options[METIS_OPTION_RTYPE] = 0; options[METIS_OPTION_UFACTOR] = 1; rval = METIS_PartGraphRecursive(numnp,&ncon,xadj,adjncy, NULL,NULL,NULL, domains,NULL,NULL,options,&edgecut,part); } else { // Kway options[METIS_OPTION_IPTYPE] = 4; options[METIS_OPTION_RTYPE] = 1; options[METIS_OPTION_UFACTOR] = 30; rval = METIS_PartGraphKway(numnp,&ncon,xadj,adjncy, NULL,NULL,NULL, domains,NULL,NULL,options,&edgecut,part); } return(rval); } From sean at mcs.anl.gov Fri Jun 8 11:04:53 2012 From: sean at mcs.anl.gov (Sean Farley) Date: Fri, 8 Jun 2012 11:04:53 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: <4FD21BD1.9020401@geology.wisc.edu> References: <4FD1F235.9080800@geology.wisc.edu> <4FD205F9.2030807@geology.wisc.edu> <4FD21BD1.9020401@geology.wisc.edu> Message-ID: > Here it is for the 3 compilers. > > GNU FC http://bpaste.net/show/31029/ > > Intel FC http://bpaste.net/show/QJ6k5jc52t7MGUlvrlEV/ > > PGI FC http://bpaste.net/show/DZzY6Re5iXMgLSHydOgJ/ You're going to need to build metis 5 with debugging options and step into the function to see why it's segfaulting. It might help (though it is a different function call) to see my mumps patch to interface with metis 5: http://petsc.cs.iit.edu/petsc/externalpackages/MUMPS_4.10.0/rev/907457aa0d14 From hanangul12 at yahoo.co.uk Fri Jun 8 11:58:43 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 8 Jun 2012 17:58:43 +0100 (BST) Subject: [petsc-users] How to implement projection preconditioner? Message-ID: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> Dear all, Summer greetings, I am back with an other query. Before I successfully implement the projection preconditioner which was simply the coarse grid correction operator P = I - A*(P*A_H*R); I implemented this simply keeping both pre and post smoothing dummy in PCMG setup. Now I need to revise this and re-implement this where I replace A and coarse operator A_H with preconditioned one i.e. M^-1 A and M^-1 A_H respectively. Thus new projection reads as P_new = I - (M^-1 A) {P*(M_H^-1 A_H)*R} Any suggestion to implement this in Petsc would be appreciated. Thanking in anticipation. with regards, Abdul -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 8 12:01:10 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 8 Jun 2012 12:01:10 -0500 Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> Message-ID: Isn't this just putting M and M_H as the preconditioning operator for the smoother? On Fri, Jun 8, 2012 at 11:58 AM, Abdul Hanan Sheikh wrote: > Dear all, > Summer greetings, > I am back with an other query. > Before I successfully implement the projection preconditioner which was > simply > the coarse grid correction operator P = I - A*(P*A_H*R); > I implemented this simply keeping both pre and post smoothing dummy in > PCMG setup. > Now I need to revise this and re-implement this where I replace A and > coarse operator A_H with > preconditioned one i.e. M^-1 A and M^-1 A_H respectively. Thus new > projection reads as > > > P_new = I - (M^-1 A) {P*(M_H^-1 A_H)*R} > > Any suggestion to implement this in Petsc would be appreciated. > > Thanking in anticipation. > with regards, Abdul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jun 8 14:35:29 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 08 Jun 2012 21:35:29 +0200 Subject: [petsc-users] Slow speed when using PETSc multigrid In-Reply-To: References: <4FCD2489.5000909@gmail.com> <5A679E25-383F-4712-9C0E-F97E84FF1938@mcs.anl.gov> <4FCFB7B7.1090304@gmail.com> <69439126-7332-4891-BF58-697966F0D975@mcs.anl.gov> <4FCFC9E4.3050301@gmail.com> <4FD059A5.3030300@gmail.com> <4FD123AB.1070009@gmail.com> Message-ID: <4FD25401.4080607@gmail.com> On 8/6/2012 12:06 AM, Jed Brown wrote: > On Thu, Jun 7, 2012 at 4:56 PM, TAY wee-beng > wrote: > > I looked at the manual but I'm not sure how to get better performance. > > * ##########################################################* > * # #* > * # WARNING!!! #* > * # #* > * # This code was compiled with a debugging option, #* > * # To get timing results run ./configure #* > * # using --with-debugging=no, the performance will #* > * # be generally two or three times faster. #* > * # #* > * ##########################################################* > > I wonder if making this text bigger, red, bold, and blinking (which I > can't seem to do in this email) would increase the chances that you > read it. Thanks for the reminder Jed, I was having segmentation fault earlier and hence I compile in debug mode to detect the error. I'll try out different options to determine the best options. > > Also, what are the more common options to start with. Is there an appropriate C example? Some options are: > > *PCMGSetLevels* - how many lvls are appropriate? > > PCMGSetCycleType - > > PCMGSetNumberSmoothUp/down etc > > 1. Control this stuff using command line options. Recompiling to run a > different method was obsolete decades ago. > > 2. You need to understand a little about the math and a little about > the performance of each component in isolation. I recommend seeing how > time moves between different operations as you modify the algorithm. > > 3. Experiment with your machine and the methods. Learn which > configurations have good algorithmic performance (few iterations) and > which execute most efficiently. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 8 17:10:50 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 8 Jun 2012 17:10:50 -0500 Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> Message-ID: <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSeqAIJWithArrays.html http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html On Jun 8, 2012, at 1:31 AM, w_ang_temp wrote: > Hello, > When solving a linear system Ax=b, the first step is assigning values to > matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove Ax=b > with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set values > to PETSc Mat A. > The variables 'values'?'columns'?'rowIndex' belong to the main function. > They are used to represent a matrix in CSR format. The following table describes > the arrays in terms of the values, row, and column positions of the non-zero > elements in a sparse matrix. > values: A real or complex array that contains the non-zero elements of a > sparse matrix. The non-zero elements are mapped into the values > array using the row-major upper triangular storage mapping described > above. > columns: Element i of the integer array columns is the number of the column > that contains the i-th element in the values array. > rowIndex: Element j of the integer array rowIndex gives the index of the > element in the values array that is first non-zero element in a row j. > > codes: > ---------Set Values to A From CSR Format(values,rowIndex,columns)------------------- > !values:Non-Symmetric Matrix > ione = 1 > do 10,II=Istart+1,Iend > !non-zero numbers of this row > rowNum=rowIndex(II+1)-rowIndex(II) > do 11,JJ=1,rowNum > > !elemnt index of the values/columns > kValNn=rowIndex(II)+JJ-1 > !column index of this element > nCol=columns(kValNn)-1 > > !Setdata:II-RowIndex,nCol-ColIndex > nRow=II-1 > call MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) > 11 continue > 10 continue > -------------------------------------------------------------------------------------- > > As we can see, it has to loop a number of times because it can only set one data per > time. And this leads to low efficiency. > So is there a better way to do this? > Thank you. > Jim > > > From w_ang_temp at 163.com Fri Jun 8 23:21:22 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sat, 9 Jun 2012 12:21:22 +0800 (CST) Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> Message-ID: <44d3cb57.3efd.137cf799b4c.Coremail.w_ang_temp@163.com> Thanks very much. And I will have a try. Jim At 2012-06-09 06:10:50,"Barry Smith" wrote: > >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSeqAIJWithArrays.html >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html > > > >On Jun 8, 2012, at 1:31 AM, w_ang_temp wrote: > >> Hello, >> When solving a linear system Ax=b, the first step is assigning values to >> matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove Ax=b >> with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set values >> to PETSc Mat A. >> The variables 'values'?'columns'?'rowIndex' belong to the main function. >> They are used to represent a matrix in CSR format. The following table describes >> the arrays in terms of the values, row, and column positions of the non-zero >> elements in a sparse matrix. >> values: A real or complex array that contains the non-zero elements of a >> sparse matrix. The non-zero elements are mapped into the values >> array using the row-major upper triangular storage mapping described >> above. >> columns: Element i of the integer array columns is the number of the column >> that contains the i-th element in the values array. >> rowIndex: Element j of the integer array rowIndex gives the index of the >> element in the values array that is first non-zero element in a row j. >> >> codes: >> ---------Set Values to A From CSR Format(values,rowIndex,columns)------------------- >> !values:Non-Symmetric Matrix >> ione = 1 >> do 10,II=Istart+1,Iend >> !non-zero numbers of this row >> rowNum=rowIndex(II+1)-rowIndex(II) >> do 11,JJ=1,rowNum >> >> !elemnt index of the values/columns >> kValNn=rowIndex(II)+JJ-1 >> !column index of this element >> nCol=columns(kValNn)-1 >> >> !Setdata:II-RowIndex,nCol-ColIndex >> nRow=II-1 >> call MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) >> 11 continue >> 10 continue >> -------------------------------------------------------------------------------------- >> >> As we can see, it has to loop a number of times because it can only set one data per >> time. And this leads to low efficiency. >> So is there a better way to do this? >> Thank you. >> Jim >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stali at geology.wisc.edu Sat Jun 9 19:23:41 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Sat, 09 Jun 2012 19:23:41 -0500 Subject: [petsc-users] METIS 5 in PETSc 3, 3 [not directly related to PETSc] In-Reply-To: References: <4FD1F235.9080800@geology.wisc.edu> <4FD205F9.2030807@geology.wisc.edu> <4FD21BD1.9020401@geology.wisc.edu> Message-ID: <4FD3E90D.8020707@geology.wisc.edu> I was passing vwgt(:)=>null() instead of vwgt=>null() and so on. The following seems to work fine on all three compilers. program test implicit none integer, parameter :: nels=2, nnds=6, npel=4 integer :: eptr(nels+1), nodes(nels*npel), epart(nels), npart(nnds), n integer, pointer :: vwgt=>null(), vsize=>null(), mopts=>null() real(8), pointer :: tpwgts=>null() eptr=(/0,4,8/) nodes=(/0,1,2,3,1,4,5,2/) call METIS_PartMeshNodal(nels,nnds,eptr,nodes,vwgt,vsize,2,tpwgts,mopts,n,epart,npart) print*, npart; print*, epart end program test Thanks for all help/suggestions. T On 06/08/2012 11:04 AM, Sean Farley wrote: >> Here it is for the 3 compilers. >> >> GNU FC http://bpaste.net/show/31029/ >> >> Intel FC http://bpaste.net/show/QJ6k5jc52t7MGUlvrlEV/ >> >> PGI FC http://bpaste.net/show/DZzY6Re5iXMgLSHydOgJ/ > You're going to need to build metis 5 with debugging options and step > into the function to see why it's segfaulting. It might help (though > it is a different function call) to see my mumps patch to interface > with metis 5: > > http://petsc.cs.iit.edu/petsc/externalpackages/MUMPS_4.10.0/rev/907457aa0d14 From w_ang_temp at 163.com Sun Jun 10 03:48:50 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 10 Jun 2012 16:48:50 +0800 (CST) Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> Message-ID: <7fbcf7a5.9794.137d594d682.Coremail.w_ang_temp@163.com> Hello, Barry I try to use MatCreateSeqAIJWithArrays to deal with my problem. Since there is no example in the doc, I find some examples in the web and use it. And I get some errors. When the processor is 1(mpiexec -n 1 ./ex4f), the results are ok. While it is more than one, I get the errors. So I think I miss some piece of information about MatCreateSeqAIJWithArrays. Thanks. Jim ----------------------------code------------------------------------------- call MatCreate(PETSC_COMM_WORLD,A,ierr) call MatSetType(A,MATAIJ,ierr) call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr) call MatSetFromOptions(A,ierr) call MatGetOwnershipRange(A,Istart,Iend,ierr) !NROWIN,NNZIJ,values:the CSR variables of the main function !NROWIN,NNZIJ:one-based,so get zero-based variables rowIndex,columns allocate(rowIndex(JFLNG+1)) allocate(columns(MAXNNZ)) rowIndex=NROWIN-1 columns=NNZIJ-1 call MatCreateSeqAIJWithArrays(MPI_COMM_WORLD,m,n,rowIndex,columns, & values,A, ierr) call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) ----------------------------code------------------------------------------- ----------------------------Error Message---------------------------------- ......... [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Mat object's type is not set: Argument # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 01:28:48 2012 [1]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatScaleSystem() line 920 in src/mat/interface/matrix.c [1]PETSC ERROR: PCPreSolve() line 1376 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPSolve() line 404 in src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Argument out of range! [1]PETSC ERROR: Comm must be of size 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 01:28:48 2012 [1]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatCreate_SeqAIJ() line 3794 in src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: MatSetType() line 73 in src/mat/interface/matreg.c [1]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4171 in src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Mat object's type is not set: Argument # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 01:28:48 2012 [1]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatAssemblyBegin() line 4760 in src/mat/interface/matrix.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Mat object's type is not set: Argument # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 01:28:48 2012 [1]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatAssemblyEnd() line 4937 in src/mat/interface/matrix.c [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Object is in wrong state! [1]PETSC ERROR: Mat object's type is not set: Argument # 1! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 01:28:48 2012 [1]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatScaleSystem() line 920 in src/mat/interface/matrix.c [1]PETSC ERROR: PCPreSolve() line 1376 in src/ksp/pc/interface/precon.c [1]PETSC ERROR: KSPSolve() line 404 in src/ksp/ksp/interface/itfunc.c ----------------------------Error Message---------------------------------- At 2012-06-09 06:10:50,"Barry Smith" wrote: > >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSeqAIJWithArrays.html >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html > > > >On Jun 8, 2012, at 1:31 AM, w_ang_temp wrote: > >> Hello, >> When solving a linear system Ax=b, the first step is assigning values to >> matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove Ax=b >> with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set values >> to PETSc Mat A. >> The variables 'values'?'columns'?'rowIndex' belong to the main function. >> They are used to represent a matrix in CSR format. The following table describes >> the arrays in terms of the values, row, and column positions of the non-zero >> elements in a sparse matrix. >> values: A real or complex array that contains the non-zero elements of a >> sparse matrix. The non-zero elements are mapped into the values >> array using the row-major upper triangular storage mapping described >> above. >> columns: Element i of the integer array columns is the number of the column >> that contains the i-th element in the values array. >> rowIndex: Element j of the integer array rowIndex gives the index of the >> element in the values array that is first non-zero element in a row j. >> >> codes: >> ---------Set Values to A From CSR Format(values,rowIndex,columns)------------------- >> !values:Non-Symmetric Matrix >> ione = 1 >> do 10,II=Istart+1,Iend >> !non-zero numbers of this row >> rowNum=rowIndex(II+1)-rowIndex(II) >> do 11,JJ=1,rowNum >> >> !elemnt index of the values/columns >> kValNn=rowIndex(II)+JJ-1 >> !column index of this element >> nCol=columns(kValNn)-1 >> >> !Setdata:II-RowIndex,nCol-ColIndex >> nRow=II-1 >> call MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) >> 11 continue >> 10 continue >> -------------------------------------------------------------------------------------- >> >> As we can see, it has to loop a number of times because it can only set one data per >> time. And this leads to low efficiency. >> So is there a better way to do this? >> Thank you. >> Jim >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Jun 10 04:42:12 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 10 Jun 2012 11:42:12 +0200 Subject: [petsc-users] Options to set when solving Ax=b if matrix A changes its non-zero position at each solve Message-ID: <4FD46BF4.6030105@gmail.com> Hi, Currently, my matrix A changes its non zero position at each time step. For e.g.: At a particular node, at time step = 1, I will insert values into its east, west neighbours. At time step = 2, I will insert values into its east, north neighbours. Hence, the non-zero positions at each row will be different at each time step. However, the max values per row will always be 5 or less. Currently, after MatSetValues and MatAssemblyBegin/End, I call: /call MatSetOption(A_mat,MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr) call KSPSetOperators(ksp,A_mat,A_mat,DIFFERENT_NONZERO_PATTERN,ierr) call KSPGetPC(ksp,pc,ierr) ksptype=KSPBCGS call KSPSetType(ksp,ksptype,ierr) call KSPSetFromOptions(ksp,ierr) call KSPSolve(ksp,b_rhs,xx,ierr)/ Is this sufficient and correct? If so, apart from KSPSolve, MatSetValues and MatAssemblyBegin/End, can I only call the other subroutines once at the 1st time step? Or do I have to explicitly destroy the matrix and create a new one? I think I ask this question a while ago but I can't find the email now. -- Yours sincerely, TAY wee-beng -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jun 10 07:42:42 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 10 Jun 2012 20:42:42 +0800 Subject: [petsc-users] Options to set when solving Ax=b if matrix A changes its non-zero position at each solve In-Reply-To: <4FD46BF4.6030105@gmail.com> References: <4FD46BF4.6030105@gmail.com> Message-ID: On Sun, Jun 10, 2012 at 5:42 PM, TAY wee-beng wrote: > Hi, > > Currently, my matrix A changes its non zero position at each time step. > > For e.g.: At a particular node, at time step = 1, I will insert values > into its east, west neighbours. At time step = 2, I will insert values > into its east, north neighbours. Hence, the non-zero positions at each row > will be different at each time step. However, the max values per row will > always be 5 or less. > > Currently, after MatSetValues and MatAssemblyBegin/End, I call: > > *call MatSetOption(A_mat,MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr) > > call KSPSetOperators(ksp,A_mat,A_mat,DIFFERENT_NONZERO_PATTERN,ierr) > > call KSPGetPC(ksp,pc,ierr) > > ksptype=KSPBCGS > > call KSPSetType(ksp,ksptype,ierr) > > call KSPSetFromOptions(ksp,ierr) > > call KSPSolve(ksp,b_rhs,xx,ierr)* > > Is this sufficient and correct? If so, apart from KSPSolve, MatSetValues > and MatAssemblyBegin/End, can I only call the other subroutines once at the > 1st time step? > > Or do I have to explicitly destroy the matrix and create a new one? I > think I ask this question a while ago but I can't find the email now. > After assembly the nonzero pattern is fixed. The right thing to do is put in explicit zeros for the rest of the stencil. Matt > > -- > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jun 10 07:49:13 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 10 Jun 2012 20:49:13 +0800 Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <7fbcf7a5.9794.137d594d682.Coremail.w_ang_temp@163.com> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> <7fbcf7a5.9794.137d594d682.Coremail.w_ang_temp@163.com> Message-ID: On Sun, Jun 10, 2012 at 4:48 PM, w_ang_temp wrote: > Hello, Barry > > I try to use MatCreateSeqAIJWithArrays to deal with my problem. Since > there is no example in the doc, I find some examples in the web and use it. > And I get some errors. > > When the processor is 1(mpiexec -n 1 ./ex4f), the results are ok. While > it is more than one, I get the errors. So I think I miss some piece of > information about MatCreateSeqAIJWithArrays. > 1) Check the error codes. Always! > Thanks. > Jim > > ----------------------------code------------------------------------------- > call MatCreate(PETSC_COMM_WORLD,A,ierr) > call MatSetType(A,MATAIJ,ierr) > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr) > call MatSetFromOptions(A,ierr) > call MatGetOwnershipRange(A,Istart,Iend,ierr) > > !NROWIN,NNZIJ,values:the CSR variables of the main function > !NROWIN,NNZIJ:one-based,so get zero-based variables rowIndex,columns > allocate(rowIndex(JFLNG+1)) > allocate(columns(MAXNNZ)) > rowIndex=NROWIN-1 > columns=NNZIJ-1 > > call MatCreateSeqAIJWithArrays(MPI_COMM_WORLD,m,n,rowIndex,columns, > & values,A, ierr) > 2) This can only be called in serial. In parallel, you need MatCreateMPIAIJWithArrays() 3) I still believe you are really doing the wrong thing. The right solution is to set one row at a time. I will bet dollars to doughnuts that the assembly time is not noticeable in your program. Matt > > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) > ----------------------------code------------------------------------------- > > ----------------------------Error Message---------------------------------- > ......... > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Mat object's type is not set: Argument # 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 > 01:28:48 2012 > [1]PETSC ERROR: Libraries linked from > /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 > [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: MatScaleSystem() line 920 in src/mat/interface/matrix.c > [1]PETSC ERROR: PCPreSolve() line 1376 in src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSolve() line 404 in src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Argument out of range! > [1]PETSC ERROR: Comm must be of size 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 > 01:28:48 2012 > [1]PETSC ERROR: Libraries linked from > /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 > [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: MatCreate_SeqAIJ() line 3794 in src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: MatSetType() line 73 in src/mat/interface/matreg.c > [1]PETSC ERROR: MatCreateSeqAIJWithArrays() line 4171 in > src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Mat object's type is not set: Argument # 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 > 01:28:48 2012 > [1]PETSC ERROR: Libraries linked from > /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 > [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: MatAssemblyBegin() line 4760 in src/mat/interface/matrix.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Mat object's type is not set: Argument # 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 > 01:28:48 2012 > [1]PETSC ERROR: Libraries linked from > /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 > [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: MatAssemblyEnd() line 4937 in src/mat/interface/matrix.c > [1]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [1]PETSC ERROR: Object is in wrong state! > [1]PETSC ERROR: Mat object's type is not set: Argument # 1! > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 > CDT 2012 > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [1]PETSC ERROR: See docs/index.html for manual pages. > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Sun Jun 10 > 01:28:48 2012 > [1]PETSC ERROR: Libraries linked from > /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-debug/lib > [1]PETSC ERROR: Configure run at Sat Jun 2 00:53:03 2012 > [1]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: MatScaleSystem() line 920 in src/mat/interface/matrix.c > [1]PETSC ERROR: PCPreSolve() line 1376 in src/ksp/pc/interface/precon.c > [1]PETSC ERROR: KSPSolve() line 404 in src/ksp/ksp/interface/itfunc.c > ----------------------------Error Message---------------------------------- > > > > At 2012-06-09 06:10:50,"Barry Smith" wrote: > > > >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSeqAIJWithArrays.html > >http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJWithArrays.html > > > > > > > >On Jun 8, 2012, at 1:31 AM, w_ang_temp wrote: > > > >> Hello, > >> When solving a linear system Ax=b, the first step is assigning values to > >> matrix A. In my program, the subroutine PETSCSOLVE, which is used to slove Ax=b > >> with PETSc, gets the CSR format matrix(values, columns, rowIndex) to set values > >> to PETSc Mat A. > >> The variables 'values'?'columns'?'rowIndex' belong to the main function. > >> They are used to represent a matrix in CSR format. The following table describes > >> the arrays in terms of the values, row, and column positions of the non-zero > >> elements in a sparse matrix. > >> values: A real or complex array that contains the non-zero elements of a > >> sparse matrix. The non-zero elements are mapped into the values > >> array using the row-major upper triangular storage mapping described > >> above. > >> columns: Element i of the integer array columns is the number of the column > >> that contains the i-th element in the values array. > >> rowIndex: Element j of the integer array rowIndex gives the index of the > >> element in the values array that is first non-zero element in a row j. > >> > >> codes: > >> ---------Set Values to A From CSR Format(values,rowIndex,columns)------------------- > >> !values:Non-Symmetric Matrix > >> ione = 1 > >> do 10,II=Istart+1,Iend > >> !non-zero numbers of this row > >> rowNum=rowIndex(II+1)-rowIndex(II) > >> do 11,JJ=1,rowNum > >> > >> !elemnt index of the values/columns > >> kValNn=rowIndex(II)+JJ-1 > >> !column index of this element > >> nCol=columns(kValNn)-1 > >> > >> !Setdata:II-RowIndex,nCol-ColIndex > >> nRow=II-1 > >> call MatSetValues(A,ione,nRow,ione,nCol,values(kValNn),INSERT_VALUES,ierr) > >> 11 continue > >> 10 continue > >> -------------------------------------------------------------------------------------- > >> > >> As we can see, it has to loop a number of times because it can only set one data per > >> time. And this leads to low efficiency. > >> So is there a better way to do this? > >> Thank you. > >> Jim > >> > >> > >> > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 00:10:56 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 01:10:56 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem Message-ID: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> Hello I am working on a Poisson Solver and ran into a problem using the multigrid preconditioner. I am using petsc-3.0.0-p12. Here's the relevant section of the code: ////////////////////Solver Stuff//////////////////// KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetType(ksp, KSPGMRES); KSPSetFromOptions(ksp); KSPGetPC(ksp, &pc); PCSetType(pc, PCMG); PCMGSetLevels(pc, 2, PETSC_NULL); PCMGSetType(pc, PC_MG_MULTIPLICATIVE); PCMGSetCycleType(pc, PC_MG_CYCLE_V); MatDuplicate(A, MAT_COPY_VALUES, &P); PCMGSetCyclesOnLevel(pc, 0, 1); PCMGSetCyclesOnLevel(pc, 1, 1); PCMGGetCoarseSolve(pc, &ksp); PCMGGetSmoother(pc, 0, &ksp); PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); PCMGSetRestriction(pc, 1, P); PCMGSetInterpolation(pc, 1, P); PCMGSetRestriction(pc, 1, P); PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); //Code breaks if I use the next line. Why? //It worked fine in the 1d case. //PCMGGetSmoother(pc, 1, &ksp); KSPSetOperators(ksp, A, P, DIFFERENT_NONZERO_PATTERN); //KSPSetOperators( ksp, A, A, SAME_NONZERO_PATTERN ); //KSPSetOperators( ksp, A, A, SAME_PRECONDITIONER ); KSPSetFromOptions(ksp); KSPSolve(ksp, breal, xreal_harm); KSPSolve(ksp, bimag, ximag_harm); /////////////////////////////////////////////////// As you can see, using the line //PCMGGetSmoother(pc, 1, &ksp); causes the solver to converge to a wrong value. What's appearing strange to me is that this exact code works for a 1 dimensional laplacian matrix (for both uniform and nonuniform grids). I don't understand why it won't work with a 2 dimensional case. Another important point is that, without the culprit line, the code converges to the right solution. Also, if I use a different ksp object and pass that object to the smoother on level 1, the code works again. I would be greatly helped if anyone can explain what is happening here. Thanks -- Nakib From jedbrown at mcs.anl.gov Sun Jun 10 09:28:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 09:28:13 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 12:10 AM, Nakib Haider Protik wrote: > I am working on a Poisson Solver and ran into a problem using the > multigrid preconditioner. I am using petsc-3.0.0-p12. > Please upgrade to petsc-3.3 > Here's the relevant > section of the code: > > ////////////////////Solver Stuff//////////////////// > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPGMRES); > > KSPSetFromOptions(ksp); > > KSPGetPC(ksp, &pc); > PCSetType(pc, PCMG); > PCMGSetLevels(pc, 2, PETSC_NULL); > PCMGSetType(pc, PC_MG_MULTIPLICATIVE); > PCMGSetCycleType(pc, PC_MG_CYCLE_V); > MatDuplicate(A, MAT_COPY_VALUES, &P); > PCMGSetCyclesOnLevel(pc, 0, 1); > PCMGSetCyclesOnLevel(pc, 1, 1); > PCMGGetCoarseSolve(pc, &ksp); > PCMGGetSmoother(pc, 0, &ksp); > PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetInterpolation(pc, 1, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); > > //Code breaks if I use the next line. Why? > //It worked fine in the 1d case. > //PCMGGetSmoother(pc, 1, &ksp); > This overwrites the "ksp" variable > > > KSPSetOperators(ksp, A, P, DIFFERENT_NONZERO_PATTERN); > //KSPSetOperators( ksp, A, A, SAME_NONZERO_PATTERN ); > //KSPSetOperators( ksp, A, A, SAME_PRECONDITIONER ); > KSPSetFromOptions(ksp); > KSPSolve(ksp, breal, xreal_harm); > KSPSolve(ksp, bimag, ximag_harm); > which you use in place of the "fine grid" problem here. It probably accidentally works in 1D because the "smoother" involved ILU which is equivalent to LU for a 1D problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Jun 10 11:09:17 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 10 Jun 2012 18:09:17 +0200 Subject: [petsc-users] Options to set when solving Ax=b if matrix A changes its non-zero position at each solve In-Reply-To: References: <4FD46BF4.6030105@gmail.com> Message-ID: <4FD4C6AD.9080906@gmail.com> On 10/6/2012 2:42 PM, Matthew Knepley wrote: > On Sun, Jun 10, 2012 at 5:42 PM, TAY wee-beng > wrote: > > Hi, > > Currently, my matrix A changes its non zero position at each time > step. > > For e.g.: At a particular node, at time step = 1, I will insert > values into its east, west neighbours. At time step = 2, I will > insert values into its east, north neighbours. Hence, the non-zero > positions at each row will be different at each time step. > However, the max values per row will always be 5 or less. > > Currently, after MatSetValues and MatAssemblyBegin/End, I call: > > /call MatSetOption(A_mat,MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr) > > call KSPSetOperators(ksp,A_mat,A_mat,DIFFERENT_NONZERO_PATTERN,ierr) > > call KSPGetPC(ksp,pc,ierr) > > ksptype=KSPBCGS > > call KSPSetType(ksp,ksptype,ierr) > > call KSPSetFromOptions(ksp,ierr) > > call KSPSolve(ksp,b_rhs,xx,ierr)/ > > Is this sufficient and correct? If so, apart from KSPSolve, > MatSetValues and MatAssemblyBegin/End, can I only call the other > subroutines once at the 1st time step? > > Or do I have to explicitly destroy the matrix and create a new > one? I think I ask this question a while ago but I can't find the > email now. > > > After assembly the nonzero pattern is fixed. The right thing to do is > put in explicit zeros for the rest of the stencil. > > Matt Hi, Just to clarify, do you mean that I create a bigger stencil and put in explicit zeros for those which I don't need at each time step? I realise that for my problem, instead of the original 5 non-zero locations (east, west, north, south, center) for the stencil, I now have 4 more (east-east, west-west etc). So now I call MATCREATEAIJ with dnz + o_nz = 9 (instead of previously 5). Is that correct? > > > -- > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 10 11:10:24 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 11:10:24 -0500 Subject: [petsc-users] Options to set when solving Ax=b if matrix A changes its non-zero position at each solve In-Reply-To: <4FD4C6AD.9080906@gmail.com> References: <4FD46BF4.6030105@gmail.com> <4FD4C6AD.9080906@gmail.com> Message-ID: On Sun, Jun 10, 2012 at 11:09 AM, TAY wee-beng wrote: > I realise that for my problem, instead of the original 5 non-zero > locations (east, west, north, south, center) for the stencil, I now have 4 > more (east-east, west-west etc). So now I call MATCREATEAIJ with dnz + o_nz > = 9 (instead of previously 5). Is that correct? yes -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 15:49:04 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 16:49:04 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> Message-ID: <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> Thank you very much for the reply. Here's a change that works for the 2D case: ////////////////////MG Solver Stuff//////////////////// KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetType(ksp, KSPGMRES); KSPCreate(PETSC_COMM_WORLD, &kspu); KSPSetType(kspu, KSPGMRES); KSPGetPC(ksp, &pc); KSPGetPC(kspu, &pc); PCSetType(pc, PCMG); PCMGSetLevels(pc, 2, PETSC_NULL); PCMGSetType(pc, PC_MG_MULTIPLICATIVE); PCMGSetCycleType(pc, PC_MG_CYCLE_V); MatDuplicate(A, MAT_COPY_VALUES, &P); PCMGSetCyclesOnLevel(pc, 0, 1); PCMGSetCyclesOnLevel(pc, 1, 1); PCMGGetCoarseSolve(pc, &ksp); PCMGGetSmootherDown(pc, 0, &ksp); PCMGGetSmootherUp(pc, 1, &kspu);//(*)// PCMGSetInterpolation(pc, 1, P); PCMGSetRestriction(pc, 1, P); PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); KSPSetFromOptions(ksp); KSPSolve(ksp, breal, xreal_harm); KSPSolve(ksp, bimag, ximag_harm); ////////////////////////////////////////////////////////// Do you think this is working again because of some accident? Without the step marked with a //(*)//, the code works too. Thanks > On Sun, Jun 10, 2012 at 12:10 AM, Nakib Haider Protik > wrote: > >> I am working on a Poisson Solver and ran into a problem using the >> multigrid preconditioner. I am using petsc-3.0.0-p12. >> > > Please upgrade to petsc-3.3 > > >> Here's the relevant >> section of the code: >> >> ////////////////////Solver Stuff//////////////////// >> KSPCreate(PETSC_COMM_WORLD, &ksp); >> KSPSetType(ksp, KSPGMRES); >> >> KSPSetFromOptions(ksp); >> >> KSPGetPC(ksp, &pc); >> PCSetType(pc, PCMG); >> PCMGSetLevels(pc, 2, PETSC_NULL); >> PCMGSetType(pc, PC_MG_MULTIPLICATIVE); >> PCMGSetCycleType(pc, PC_MG_CYCLE_V); >> MatDuplicate(A, MAT_COPY_VALUES, &P); >> PCMGSetCyclesOnLevel(pc, 0, 1); >> PCMGSetCyclesOnLevel(pc, 1, 1); >> PCMGGetCoarseSolve(pc, &ksp); >> PCMGGetSmoother(pc, 0, &ksp); >> PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); >> PCMGSetRestriction(pc, 1, P); >> PCMGSetInterpolation(pc, 1, P); >> PCMGSetRestriction(pc, 1, P); >> PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); >> >> //Code breaks if I use the next line. Why? >> //It worked fine in the 1d case. >> //PCMGGetSmoother(pc, 1, &ksp); >> > > This overwrites the "ksp" variable > > >> >> >> KSPSetOperators(ksp, A, P, DIFFERENT_NONZERO_PATTERN); >> //KSPSetOperators( ksp, A, A, SAME_NONZERO_PATTERN ); >> //KSPSetOperators( ksp, A, A, SAME_PRECONDITIONER ); >> KSPSetFromOptions(ksp); >> KSPSolve(ksp, breal, xreal_harm); >> KSPSolve(ksp, bimag, ximag_harm); >> > > which you use in place of the "fine grid" problem here. > > It probably accidentally works in 1D because the "smoother" involved ILU > which is equivalent to LU for a 1D problem. > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 15:57:44 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 15:57:44 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 3:49 PM, Nakib Haider Protik wrote: > Thank you very much for the reply. Here's a change that works for the 2D > case: > > ////////////////////MG Solver Stuff//////////////////// > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPGMRES); > > KSPCreate(PETSC_COMM_WORLD, &kspu); > KSPSetType(kspu, KSPGMRES); > Why do you have two KSPs here? > > KSPGetPC(ksp, &pc); > KSPGetPC(kspu, &pc); > Why? > > PCSetType(pc, PCMG); > PCMGSetLevels(pc, 2, PETSC_NULL); > PCMGSetType(pc, PC_MG_MULTIPLICATIVE); > PCMGSetCycleType(pc, PC_MG_CYCLE_V); > MatDuplicate(A, MAT_COPY_VALUES, &P); > PCMGSetCyclesOnLevel(pc, 0, 1); > PCMGSetCyclesOnLevel(pc, 1, 1); > PCMGGetCoarseSolve(pc, &ksp); > PCMGGetSmootherDown(pc, 0, &ksp); > PCMGGetSmootherUp(pc, 1, &kspu);//(*)// > Okay, clearly you need to learn how variables work in C. Here is a free resource. http://c.learncodethehardway.org/book/ (Any other resource will also do.) When you are comfortable with the language, we can revisit your code. > PCMGSetInterpolation(pc, 1, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); > PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); > KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); > KSPSetFromOptions(ksp); > KSPSolve(ksp, breal, xreal_harm); > KSPSolve(ksp, bimag, ximag_harm); > ////////////////////////////////////////////////////////// > > Do you think this is working again because of some accident? Without the > step marked with a //(*)//, the code works too. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 16:20:06 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 17:20:06 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> Message-ID: <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> Reason for two ksp objects: Multigrid and Domain Decomposition in PETSc Barry Smith PETSc Developer pg 28: //Use same pre and post smoother MGGetSmoother(pc,int level,KSP *ksp); //Use different pre and post smoother MGGetSmootherDown(pc,int level,KSP *dksp); MGGetSmootherUp(pc,int level,KSP *uksp); I was trying to use the second in my code. I would be greatly helped if you could tell me where exactly my code goes wrong. As for the "accidental" correctness of the 1d code, here's where I found the prototype: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html If my question does not abide by the regulations of this mailing list, or if you are not obliged to answers questions of this kind, please let me know. I am not looking for free lessons on c, but thanks for the link. Despite the rude reply, thank you for taking the time. > On Sun, Jun 10, 2012 at 3:49 PM, Nakib Haider Protik > wrote: > >> Thank you very much for the reply. Here's a change that works for the 2D >> case: >> >> ////////////////////MG Solver Stuff//////////////////// >> KSPCreate(PETSC_COMM_WORLD, &ksp); >> KSPSetType(ksp, KSPGMRES); >> >> KSPCreate(PETSC_COMM_WORLD, &kspu); >> KSPSetType(kspu, KSPGMRES); >> > > Why do you have two KSPs here? > > >> >> KSPGetPC(ksp, &pc); >> KSPGetPC(kspu, &pc); >> > > Why? > > >> >> PCSetType(pc, PCMG); >> PCMGSetLevels(pc, 2, PETSC_NULL); >> PCMGSetType(pc, PC_MG_MULTIPLICATIVE); >> PCMGSetCycleType(pc, PC_MG_CYCLE_V); >> MatDuplicate(A, MAT_COPY_VALUES, &P); >> PCMGSetCyclesOnLevel(pc, 0, 1); >> PCMGSetCyclesOnLevel(pc, 1, 1); >> PCMGGetCoarseSolve(pc, &ksp); >> PCMGGetSmootherDown(pc, 0, &ksp); >> PCMGGetSmootherUp(pc, 1, &kspu);//(*)// >> > > Okay, clearly you need to learn how variables work in C. Here is a free > resource. > > http://c.learncodethehardway.org/book/ > > (Any other resource will also do.) When you are comfortable with the > language, we can revisit your code. > > >> PCMGSetInterpolation(pc, 1, P); >> PCMGSetRestriction(pc, 1, P); >> PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); >> PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); >> KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); >> KSPSetFromOptions(ksp); >> KSPSolve(ksp, breal, xreal_harm); >> KSPSolve(ksp, bimag, ximag_harm); >> ////////////////////////////////////////////////////////// >> >> Do you think this is working again because of some accident? Without the >> step marked with a //(*)//, the code works too. >> > -- Nakib From jedbrown at mcs.anl.gov Sun Jun 10 16:33:28 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 16:33:28 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 4:20 PM, Nakib Haider Protik wrote: > Reason for two ksp objects: > > Multigrid and Domain Decomposition in PETSc > Barry Smith > PETSc Developer > > pg 28: > > //Use same pre and post smoother > MGGetSmoother(pc,int level,KSP *ksp); > > > //Use different pre and post smoother > MGGetSmootherDown(pc,int level,KSP *dksp); > MGGetSmootherUp(pc,int level,KSP *uksp); > > I was trying to use the second in my code. > But you created two outer-level KSP objects, then leaked their memory when you overwrote their value when accessing the smoother on levels. > I would be greatly helped if > you could tell me where exactly my code goes wrong. > The problem is that you are using the same local variable for every ksp arising in the code. When you get access to levels, you are overwriting the local variable. Since you did this so many times in various places where it was clearly not intended, I had assumed that you were not familiar with local variables in C. Everything that was incorrect with your code had to do with overwriting the local variable and later using it as though its original value was still intact. It would be better to explain what you are trying to achieve instead of posting code with nonsensical handling of local variables. You should read through the code you posted and fix all these places (declare new variables of type KSP so you don't overwrite the outer ones). If you still have trouble, run in valgrind. It will tell you when you lose the reference to the outer objects. > > As for the "accidental" correctness of the 1d code, here's where I found > the prototype: > > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html > > If my question does not abide by the regulations of this mailing list, or > if you are not obliged to answers questions of this kind, please let me > know. I am not looking for free lessons on c, but thanks for the link. > Despite the rude reply, thank you for taking the time. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 16:39:28 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 17:39:28 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> Message-ID: <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> Thank you for the fast reply. Here's where I found a pcmg implementation that "looks more or less correct". Does this code suffer from the same problem you were referring to? Many thanks in advance. > On Sun, Jun 10, 2012 at 4:20 PM, Nakib Haider Protik > wrote: > >> Reason for two ksp objects: >> >> Multigrid and Domain Decomposition in PETSc >> Barry Smith >> PETSc Developer >> >> pg 28: >> >> //Use same pre and post smoother >> MGGetSmoother(pc,int level,KSP *ksp); >> >> >> //Use different pre and post smoother >> MGGetSmootherDown(pc,int level,KSP *dksp); >> MGGetSmootherUp(pc,int level,KSP *uksp); >> >> I was trying to use the second in my code. >> > > But you created two outer-level KSP objects, then leaked their memory when > you overwrote their value when accessing the smoother on levels. > > >> I would be greatly helped if >> you could tell me where exactly my code goes wrong. >> > > The problem is that you are using the same local variable for every ksp > arising in the code. When you get access to levels, you are overwriting > the > local variable. Since you did this so many times in various places where > it > was clearly not intended, I had assumed that you were not familiar with > local variables in C. Everything that was incorrect with your code had to > do with overwriting the local variable and later using it as though its > original value was still intact. It would be better to explain what you > are > trying to achieve instead of posting code with nonsensical handling of > local variables. > > You should read through the code you posted and fix all these places > (declare new variables of type KSP so you don't overwrite the outer ones). > If you still have trouble, run in valgrind. It will tell you when you lose > the reference to the outer objects. > > >> >> As for the "accidental" correctness of the 1d code, here's where I found >> the prototype: >> >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html >> >> If my question does not abide by the regulations of this mailing list, >> or >> if you are not obliged to answers questions of this kind, please let me >> know. I am not looking for free lessons on c, but thanks for the link. >> Despite the rude reply, thank you for taking the time. >> > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 16:44:31 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 16:44:31 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 4:39 PM, Nakib Haider Protik wrote: > Here's where I found a pcmg implementation that "looks more or less > correct". Does this code suffer from the same problem you were referring > to? > > Many thanks in advance. > Did you mean to include code or are you citing that email (which is just code snippets and seems to have the same problem of needlessly accessing various contexts and rampantly overwriting the local variable)? Please make your code leak-free according to valgrind (because it will catch all this misuse if it isn't obvious to you when looking). Look at the output of -ksp_view to see if the method is what you intend. If something doesn't match what you expect, please reply with the entire setup code, explain what goes wrong, and explain what you expect to happen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 16:54:48 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 17:54:48 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> Message-ID: <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> Sorry, I meant to refer to this: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html Thanks. > On Sun, Jun 10, 2012 at 4:39 PM, Nakib Haider Protik > wrote: > >> Here's where I found a pcmg implementation that "looks more or less >> correct". Does this code suffer from the same problem you were referring >> to? >> >> Many thanks in advance. >> > > Did you mean to include code or are you citing that email (which is just > code snippets and seems to have the same problem of needlessly accessing > various contexts and rampantly overwriting the local variable)? > > Please make your code leak-free according to valgrind (because it will > catch all this misuse if it isn't obvious to you when looking). Look at > the > output of -ksp_view to see if the method is what you intend. If something > doesn't match what you expect, please reply with the entire setup code, > explain what goes wrong, and explain what you expect to happen. > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 17:03:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 17:03:42 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 4:54 PM, Nakib Haider Protik wrote: > Sorry, I meant to refer to this: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html As I've said in every message, this rampantly overwrites local variables. > > DAGetMatrix(da, MATAIJ, &M); > > > > KSPCreate(PETSC_COMM_WORLD, &ksp); > > KSPSetType(ksp, KSPGMRES); > > KSPGetPC(ksp, &pcmg); > > PCSetType(pcmg, PCMG); > > > > PCMGSetLevels(pcmg, 2, &PETSC_COMM_WORLD); > > PCMGSetType(pcmg, PC_MG_MULTIPLICATIVE); > > PCMGSetCycleType(pcmg, PC_MG_CYCLE_W); > > PCMGSetCyclesOnLevel(pcmg, 0, 1); > > PCMGSetCyclesOnLevel(pcmg, 1, 1); > > > > PCMGGetCoarseSolve(pcmg, &ksp); This overwrites ksp with the coarse solver. > > > > PCMGGetSmoother(pcmg, 0, &ksp); This overwrites it again with the level 0 "smoother" (same as the coarse solver). > > PCMGGetSmoother(pcmg, 1, &ksp); This overwrites it again with the level 1 smoother. > > PCMGSetInterpolation(pcmg, 1, M); > > PCMGSetRestriction(pcmg, 1, M); > > > > PCMGSetResidual(pcmg, 0, PCMGDefaultResidual, M); > > PCMGSetResidual(pcmg, 1, PCMGDefaultResidual, M); So if you get down here and use "ksp" for something (as your code did), your are actually working with the level 1 smoother. I also said this in my first email, explaining why the problem was solved this way (the smoother was as good as a direct solve). The original KSP has been lost forever and has leaked its memory. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 17:08:11 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 18:08:11 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> Message-ID: <52510.99.254.3.78.1339366091.squirrel@webmail01.uottawa.ca> Thanks a million for the clarification. > On Sun, Jun 10, 2012 at 4:54 PM, Nakib Haider Protik > wrote: > >> Sorry, I meant to refer to this: >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html > > > As I've said in every message, this rampantly overwrites local variables. > >> > DAGetMatrix(da, MATAIJ, &M); >> > >> > KSPCreate(PETSC_COMM_WORLD, &ksp); >> > KSPSetType(ksp, KSPGMRES); >> > KSPGetPC(ksp, &pcmg); >> > PCSetType(pcmg, PCMG); >> > >> > PCMGSetLevels(pcmg, 2, &PETSC_COMM_WORLD); >> > PCMGSetType(pcmg, PC_MG_MULTIPLICATIVE); >> > PCMGSetCycleType(pcmg, PC_MG_CYCLE_W); >> > PCMGSetCyclesOnLevel(pcmg, 0, 1); >> > PCMGSetCyclesOnLevel(pcmg, 1, 1); >> > >> > PCMGGetCoarseSolve(pcmg, &ksp); > > This overwrites ksp with the coarse solver. > >> > >> > PCMGGetSmoother(pcmg, 0, &ksp); > > This overwrites it again with the level 0 "smoother" (same as the coarse > solver). > >> > PCMGGetSmoother(pcmg, 1, &ksp); > > This overwrites it again with the level 1 smoother. > >> > PCMGSetInterpolation(pcmg, 1, M); >> > PCMGSetRestriction(pcmg, 1, M); >> > >> > PCMGSetResidual(pcmg, 0, PCMGDefaultResidual, M); >> > PCMGSetResidual(pcmg, 1, PCMGDefaultResidual, M); > > So if you get down here and use "ksp" for something (as your code did), > your are actually working with the level 1 smoother. I also said this in > my > first email, explaining why the problem was solved this way (the smoother > was as good as a direct solve). The original KSP has been lost forever and > has leaked its memory. > -- Nakib :) From nprot048 at uottawa.ca Sun Jun 10 19:05:44 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 20:05:44 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> Message-ID: <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> Hello again Here I am trying to declare three different local KSP objects for the coarse solver, up smoother and down smoother respectively. The code fails to compile with the error: expected expression before ?KSP? for each of the /**/ lines. Both the v3.0.0 user manual and the online documentation suggests usage of this kind. I am totally at loss here and any suggestion will be greatly appreciated. ////////////////////MG Solver Stuff//////////////////// KSP solver; PC pc; Mat A, P; // . . . Here the A matrix forming function is called . . // KSPCreate(PETSC_COMM_WORLD, &solver); KSPSetType(solver, KSPGMRES); KSPGetPC(solver, &pc); KSPSetFromOptions(solver); PCSetType(pc, PCMG); PCMGSetLevels(pc, 2, PETSC_NULL); PCMGSetType(pc, PC_MG_MULTIPLICATIVE); PCMGSetCycleType(pc, PC_MG_CYCLE_V); MatDuplicate(A, MAT_COPY_VALUES, &P); PCMGSetCyclesOnLevel(pc, 0, 1); PCMGSetCyclesOnLevel(pc, 1, 1); PCMGGetCoarseSolve(pc, KSP *cksp);/**/ PCMGGetSmootherDown(pc, 0, KSP *dksp);/**/ PCMGGetSmootherUp(pc, 1, KSP *uksp);/**/ PCMGSetInterpolation(pc, 1, P); PCMGSetRestriction(pc, 1, P); PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); KSPSetOperators(solver, A, P, SAME_NONZERO_PATTERN); KSPSolve(solver, breal, xreal_harm); KSPSolve(solver, bimag, ximag_harm); ////////////////////////////////////////////////////////// Thanks > On Sun, Jun 10, 2012 at 4:54 PM, Nakib Haider Protik > wrote: > >> Sorry, I meant to refer to this: >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html > > > As I've said in every message, this rampantly overwrites local variables. > >> > DAGetMatrix(da, MATAIJ, &M); >> > >> > KSPCreate(PETSC_COMM_WORLD, &ksp); >> > KSPSetType(ksp, KSPGMRES); >> > KSPGetPC(ksp, &pcmg); >> > PCSetType(pcmg, PCMG); >> > >> > PCMGSetLevels(pcmg, 2, &PETSC_COMM_WORLD); >> > PCMGSetType(pcmg, PC_MG_MULTIPLICATIVE); >> > PCMGSetCycleType(pcmg, PC_MG_CYCLE_W); >> > PCMGSetCyclesOnLevel(pcmg, 0, 1); >> > PCMGSetCyclesOnLevel(pcmg, 1, 1); >> > >> > PCMGGetCoarseSolve(pcmg, &ksp); > > This overwrites ksp with the coarse solver. > >> > >> > PCMGGetSmoother(pcmg, 0, &ksp); > > This overwrites it again with the level 0 "smoother" (same as the coarse > solver). > >> > PCMGGetSmoother(pcmg, 1, &ksp); > > This overwrites it again with the level 1 smoother. > >> > PCMGSetInterpolation(pcmg, 1, M); >> > PCMGSetRestriction(pcmg, 1, M); >> > >> > PCMGSetResidual(pcmg, 0, PCMGDefaultResidual, M); >> > PCMGSetResidual(pcmg, 1, PCMGDefaultResidual, M); > > So if you get down here and use "ksp" for something (as your code did), > your are actually working with the level 1 smoother. I also said this in > my > first email, explaining why the problem was solved this way (the smoother > was as good as a direct solve). The original KSP has been lost forever and > has leaked its memory. > -- Nakib From jedbrown at mcs.anl.gov Sun Jun 10 19:11:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 19:11:27 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 7:05 PM, Nakib Haider Protik wrote: > Hello again > > Here I am trying to declare three different local KSP objects for the > coarse solver, up smoother and down smoother respectively. The code fails > to compile with the error: > > expected expression before ?KSP? > > for each of the /**/ lines. Both the v3.0.0 This is unrelated, but please upgrade to petsc-3.3 > user manual and the online > documentation suggests usage of this kind. I am totally at loss here and > any suggestion will be greatly appreciated. > > ////////////////////MG Solver Stuff//////////////////// > KSP solver; > PC pc; > Mat A, P; > // > . > . > . Here the A matrix forming function is called > . > . > // > KSPCreate(PETSC_COMM_WORLD, &solver); > KSPSetType(solver, KSPGMRES); > KSPGetPC(solver, &pc); > KSPSetFromOptions(solver); > > PCSetType(pc, PCMG); > PCMGSetLevels(pc, 2, PETSC_NULL); > PCMGSetType(pc, PC_MG_MULTIPLICATIVE); > PCMGSetCycleType(pc, PC_MG_CYCLE_V); > MatDuplicate(A, MAT_COPY_VALUES, &P); > PCMGSetCyclesOnLevel(pc, 0, 1); > PCMGSetCyclesOnLevel(pc, 1, 1); > PCMGGetCoarseSolve(pc, KSP *cksp);/**/ > I'm sorry, this is not valid C. The API documentation in the user's manual shows you the types in the call, it is not valid code to call the routine. I'm afraid that if you are going to write code in C, you will have to learn the C language. It is not a difficult language. > PCMGGetSmootherDown(pc, 0, KSP *dksp);/**/ > PCMGGetSmootherUp(pc, 1, KSP *uksp);/**/ > PCMGSetInterpolation(pc, 1, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); > PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); > > KSPSetOperators(solver, A, P, SAME_NONZERO_PATTERN); > KSPSolve(solver, breal, xreal_harm); > KSPSolve(solver, bimag, ximag_harm); > ////////////////////////////////////////////////////////// > > Thanks > > > On Sun, Jun 10, 2012 at 4:54 PM, Nakib Haider Protik > > wrote: > > > >> Sorry, I meant to refer to this: > >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html > > > > > > As I've said in every message, this rampantly overwrites local variables. > > > >> > DAGetMatrix(da, MATAIJ, &M); > >> > > >> > KSPCreate(PETSC_COMM_WORLD, &ksp); > >> > KSPSetType(ksp, KSPGMRES); > >> > KSPGetPC(ksp, &pcmg); > >> > PCSetType(pcmg, PCMG); > >> > > >> > PCMGSetLevels(pcmg, 2, &PETSC_COMM_WORLD); > >> > PCMGSetType(pcmg, PC_MG_MULTIPLICATIVE); > >> > PCMGSetCycleType(pcmg, PC_MG_CYCLE_W); > >> > PCMGSetCyclesOnLevel(pcmg, 0, 1); > >> > PCMGSetCyclesOnLevel(pcmg, 1, 1); > >> > > >> > PCMGGetCoarseSolve(pcmg, &ksp); > > > > This overwrites ksp with the coarse solver. > > > >> > > >> > PCMGGetSmoother(pcmg, 0, &ksp); > > > > This overwrites it again with the level 0 "smoother" (same as the coarse > > solver). > > > >> > PCMGGetSmoother(pcmg, 1, &ksp); > > > > This overwrites it again with the level 1 smoother. > > > >> > PCMGSetInterpolation(pcmg, 1, M); > >> > PCMGSetRestriction(pcmg, 1, M); > >> > > >> > PCMGSetResidual(pcmg, 0, PCMGDefaultResidual, M); > >> > PCMGSetResidual(pcmg, 1, PCMGDefaultResidual, M); > > > > So if you get down here and use "ksp" for something (as your code did), > > your are actually working with the level 1 smoother. I also said this in > > my > > first email, explaining why the problem was solved this way (the smoother > > was as good as a direct solve). The original KSP has been lost forever > and > > has leaked its memory. > > > > > -- > Nakib > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 19:32:46 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 20:32:46 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> Message-ID: <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> Okay. But if I understood correctly, doing the following would also be wrong since I will have made three extra outer KSP objects. I am not sure I understand what these functions want. KSP cksp, uksp, dksp; PCMGGetCoarseSolve(pc, &cksp);/**/ PCMGGetSmootherDown(pc, 0, &dksp);/**/ PCMGGetSmootherUp(pc, 1, &uksp);/**/ Thanks. > On Sun, Jun 10, 2012 at 7:05 PM, Nakib Haider Protik > wrote: > >> Hello again >> >> Here I am trying to declare three different local KSP objects for the >> coarse solver, up smoother and down smoother respectively. The code >> fails >> to compile with the error: >> >> expected expression before ?KSP? >> >> for each of the /**/ lines. Both the v3.0.0 > > > This is unrelated, but please upgrade to petsc-3.3 > > >> user manual and the online >> documentation suggests usage of this kind. I am totally at loss here and >> any suggestion will be greatly appreciated. >> >> ////////////////////MG Solver Stuff//////////////////// >> KSP solver; >> PC pc; >> Mat A, P; >> // >> . >> . >> . Here the A matrix forming function is called >> . >> . >> // >> KSPCreate(PETSC_COMM_WORLD, &solver); >> KSPSetType(solver, KSPGMRES); >> KSPGetPC(solver, &pc); >> KSPSetFromOptions(solver); >> >> PCSetType(pc, PCMG); >> PCMGSetLevels(pc, 2, PETSC_NULL); >> PCMGSetType(pc, PC_MG_MULTIPLICATIVE); >> PCMGSetCycleType(pc, PC_MG_CYCLE_V); >> MatDuplicate(A, MAT_COPY_VALUES, &P); >> PCMGSetCyclesOnLevel(pc, 0, 1); >> PCMGSetCyclesOnLevel(pc, 1, 1); >> PCMGGetCoarseSolve(pc, KSP *cksp);/**/ >> > > I'm sorry, this is not valid C. The API documentation in the user's manual > shows you the types in the call, it is not valid code to call the routine. > > I'm afraid that if you are going to write code in C, you will have to > learn > the C language. It is not a difficult language. > > >> PCMGGetSmootherDown(pc, 0, KSP *dksp);/**/ >> PCMGGetSmootherUp(pc, 1, KSP *uksp);/**/ >> PCMGSetInterpolation(pc, 1, P); >> PCMGSetRestriction(pc, 1, P); >> PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); >> PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); >> >> KSPSetOperators(solver, A, P, SAME_NONZERO_PATTERN); >> KSPSolve(solver, breal, xreal_harm); >> KSPSolve(solver, bimag, ximag_harm); >> ////////////////////////////////////////////////////////// >> >> Thanks >> >> > On Sun, Jun 10, 2012 at 4:54 PM, Nakib Haider Protik >> > wrote: >> > >> >> Sorry, I meant to refer to this: >> >> http://lists.mcs.anl.gov/pipermail/petsc-users/2011-May/008793.html >> > >> > >> > As I've said in every message, this rampantly overwrites local >> variables. >> > >> >> > DAGetMatrix(da, MATAIJ, &M); >> >> > >> >> > KSPCreate(PETSC_COMM_WORLD, &ksp); >> >> > KSPSetType(ksp, KSPGMRES); >> >> > KSPGetPC(ksp, &pcmg); >> >> > PCSetType(pcmg, PCMG); >> >> > >> >> > PCMGSetLevels(pcmg, 2, &PETSC_COMM_WORLD); >> >> > PCMGSetType(pcmg, PC_MG_MULTIPLICATIVE); >> >> > PCMGSetCycleType(pcmg, PC_MG_CYCLE_W); >> >> > PCMGSetCyclesOnLevel(pcmg, 0, 1); >> >> > PCMGSetCyclesOnLevel(pcmg, 1, 1); >> >> > >> >> > PCMGGetCoarseSolve(pcmg, &ksp); >> > >> > This overwrites ksp with the coarse solver. >> > >> >> > >> >> > PCMGGetSmoother(pcmg, 0, &ksp); >> > >> > This overwrites it again with the level 0 "smoother" (same as the >> coarse >> > solver). >> > >> >> > PCMGGetSmoother(pcmg, 1, &ksp); >> > >> > This overwrites it again with the level 1 smoother. >> > >> >> > PCMGSetInterpolation(pcmg, 1, M); >> >> > PCMGSetRestriction(pcmg, 1, M); >> >> > >> >> > PCMGSetResidual(pcmg, 0, PCMGDefaultResidual, M); >> >> > PCMGSetResidual(pcmg, 1, PCMGDefaultResidual, M); >> > >> > So if you get down here and use "ksp" for something (as your code >> did), >> > your are actually working with the level 1 smoother. I also said this >> in >> > my >> > first email, explaining why the problem was solved this way (the >> smoother >> > was as good as a direct solve). The original KSP has been lost forever >> and >> > has leaked its memory. >> > >> >> >> -- >> Nakib >> > -- Nakib From jedbrown at mcs.anl.gov Sun Jun 10 19:35:12 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 19:35:12 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 7:32 PM, Nakib Haider Protik wrote: > Okay. But if I understood correctly, doing the following would also be > wrong since I will have made three extra outer KSP objects. I am not sure > I understand what these functions want. > > KSP cksp, uksp, dksp; > PCMGGetCoarseSolve(pc, &cksp);/**/ > PCMGGetSmootherDown(pc, 0, &dksp);/**/ > PCMGGetSmootherUp(pc, 1, &uksp);/**/ > This is fine, all of these are getting references to inner objects without overwriting the outer objects. Of course you don't need to get these references if you aren't going to modify them. I recommend that you experiment with algorithms using the command line and check that it does what you expect using -ksp_view. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 19:54:27 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 20:54:27 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> Message-ID: <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> Okay, so here is the code that works. This gives the correct result, but I want to make sure that it is not giving the correct result accidentally. Thank you very much for your time. KSPCreate(PETSC_COMM_WORLD, &ksp); KSPCreate(PETSC_COMM_WORLD, &cksp); KSPCreate(PETSC_COMM_WORLD, &uksp); KSPCreate(PETSC_COMM_WORLD, &dksp); KSPSetType(ksp, KSPGMRES); KSPSetType(cksp, KSPGMRES); KSPSetType(uksp, KSPGMRES); KSPSetType(dksp, KSPGMRES); KSPGetPC(ksp, &pc); KSPGetPC(cksp, &pc); KSPGetPC(uksp, &pc); KSPGetPC(dksp, &pc); KSPSetFromOptions(ksp); PCSetType(pc, PCMG); PCMGSetLevels(pc, 2, PETSC_NULL); PCMGSetType(pc, PC_MG_MULTIPLICATIVE); PCMGSetCycleType(pc, PC_MG_CYCLE_V); MatDuplicate(A, MAT_COPY_VALUES, &P); PCMGSetCyclesOnLevel(pc, 0, 1); PCMGSetCyclesOnLevel(pc, 1, 1); PCMGGetCoarseSolve(pc, &cksp); PCMGGetSmootherDown(pc, 0, &dksp); PCMGGetSmootherUp(pc, 1, &uksp); PCMGSetInterpolation(pc, 1, P); PCMGSetRestriction(pc, 1, P); PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); KSPSolve(ksp, breal, xreal_harm); KSPSolve(ksp, bimag, ximag_harm); ////////////////////////////////////////////////////////// > On Sun, Jun 10, 2012 at 7:32 PM, Nakib Haider Protik > wrote: > >> Okay. But if I understood correctly, doing the following would also be >> wrong since I will have made three extra outer KSP objects. I am not >> sure >> I understand what these functions want. >> >> KSP cksp, uksp, dksp; >> PCMGGetCoarseSolve(pc, &cksp);/**/ >> PCMGGetSmootherDown(pc, 0, &dksp);/**/ >> PCMGGetSmootherUp(pc, 1, &uksp);/**/ >> > > This is fine, all of these are getting references to inner objects without > overwriting the outer objects. Of course you don't need to get these > references if you aren't going to modify them. > > I recommend that you experiment with algorithms using the command line and > check that it does what you expect using -ksp_view. > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 20:05:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 20:05:43 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 7:54 PM, Nakib Haider Protik wrote: > Okay, so here is the code that works. This gives the correct result, but I > want to make sure that it is not giving the correct result accidentally. > 1. I don't know how to emphasize this enough: you need to learn C to write C code. 2. Please run in Valgrind, it will show you several memory leaks here. 3. Please run with -ksp_view so you can see what gets used. If you did this, it would be obvious that multigrid was not being used. > Thank you very much for your time. > > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPCreate(PETSC_COMM_WORLD, &cksp); > KSPCreate(PETSC_COMM_WORLD, &uksp); > KSPCreate(PETSC_COMM_WORLD, &dksp); > ^^ You only need to create ksp here, delete the other lines > KSPSetType(ksp, KSPGMRES); > KSPSetType(cksp, KSPGMRES); > KSPSetType(uksp, KSPGMRES); > KSPSetType(dksp, KSPGMRES); > ^^ You don't need any of these > > KSPGetPC(ksp, &pc); > KSPGetPC(cksp, &pc); > KSPGetPC(uksp, &pc); > KSPGetPC(dksp, &pc); > ^^ delete all but the first of these > > KSPSetFromOptions(ksp); > > PCSetType(pc, PCMG); > PCMGSetLevels(pc, 2, PETSC_NULL); > PCMGSetType(pc, PC_MG_MULTIPLICATIVE); > PCMGSetCycleType(pc, PC_MG_CYCLE_V); > MatDuplicate(A, MAT_COPY_VALUES, &P); > PCMGSetCyclesOnLevel(pc, 0, 1); > PCMGSetCyclesOnLevel(pc, 1, 1); > > PCMGGetCoarseSolve(pc, &cksp); > PCMGGetSmootherDown(pc, 0, &dksp); > PCMGGetSmootherUp(pc, 1, &uksp); > ^^^ delete the last three lines here, they aren't doing anything > > PCMGSetInterpolation(pc, 1, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); > PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); > There is no way that you want to be using P for interpolation/restriction and as the residual on levels and as the preconditioning matrix on the line below. The matrices are not even the right size. The reason the above runs is because nothing you have done with "pc" is being used: it refers to a "dksp" which you are leaking without using. > > KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); > KSPSolve(ksp, breal, xreal_harm); > KSPSolve(ksp, bimag, ximag_harm); > ////////////////////////////////////////////////////////// > > > On Sun, Jun 10, 2012 at 7:32 PM, Nakib Haider Protik > > wrote: > > > >> Okay. But if I understood correctly, doing the following would also be > >> wrong since I will have made three extra outer KSP objects. I am not > >> sure > >> I understand what these functions want. > >> > >> KSP cksp, uksp, dksp; > >> PCMGGetCoarseSolve(pc, &cksp);/**/ > >> PCMGGetSmootherDown(pc, 0, &dksp);/**/ > >> PCMGGetSmootherUp(pc, 1, &uksp);/**/ > >> > > > > This is fine, all of these are getting references to inner objects > without > > overwriting the outer objects. Of course you don't need to get these > > references if you aren't going to modify them. > > > > I recommend that you experiment with algorithms using the command line > and > > check that it does what you expect using -ksp_view. > > > > > -- > Nakib :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 20:19:37 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 21:19:37 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> Message-ID: <54026.99.254.3.78.1339377577.squirrel@webmail01.uottawa.ca> With the suggested changes, the code doesn't run. Is there any example code of KSP solvers using multigrid preconditioning? All the examples of multigrid I have come across involve dmmg. Thanks. > On Sun, Jun 10, 2012 at 7:54 PM, Nakib Haider Protik > wrote: > >> Okay, so here is the code that works. This gives the correct result, but >> I >> want to make sure that it is not giving the correct result accidentally. >> > > 1. I don't know how to emphasize this enough: you need to learn C to write > C code. > > 2. Please run in Valgrind, it will show you several memory leaks here. > > 3. Please run with -ksp_view so you can see what gets used. If you did > this, it would be obvious that multigrid was not being used. > > >> Thank you very much for your time. >> >> KSPCreate(PETSC_COMM_WORLD, &ksp); >> KSPCreate(PETSC_COMM_WORLD, &cksp); >> KSPCreate(PETSC_COMM_WORLD, &uksp); >> KSPCreate(PETSC_COMM_WORLD, &dksp); >> > > ^^ You only need to create ksp here, delete the other lines > > >> KSPSetType(ksp, KSPGMRES); >> KSPSetType(cksp, KSPGMRES); >> KSPSetType(uksp, KSPGMRES); >> KSPSetType(dksp, KSPGMRES); >> > > ^^ You don't need any of these > > >> >> KSPGetPC(ksp, &pc); >> KSPGetPC(cksp, &pc); >> KSPGetPC(uksp, &pc); >> KSPGetPC(dksp, &pc); >> > > ^^ delete all but the first of these > > >> >> KSPSetFromOptions(ksp); >> >> PCSetType(pc, PCMG); >> PCMGSetLevels(pc, 2, PETSC_NULL); >> PCMGSetType(pc, PC_MG_MULTIPLICATIVE); >> PCMGSetCycleType(pc, PC_MG_CYCLE_V); >> MatDuplicate(A, MAT_COPY_VALUES, &P); >> PCMGSetCyclesOnLevel(pc, 0, 1); >> PCMGSetCyclesOnLevel(pc, 1, 1); >> >> PCMGGetCoarseSolve(pc, &cksp); >> PCMGGetSmootherDown(pc, 0, &dksp); >> PCMGGetSmootherUp(pc, 1, &uksp); >> > > ^^^ delete the last three lines here, they aren't doing anything > > >> >> PCMGSetInterpolation(pc, 1, P); >> PCMGSetRestriction(pc, 1, P); >> PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); >> PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); >> > > There is no way that you want to be using P for interpolation/restriction > and as the residual on levels and as the preconditioning matrix on the > line > below. The matrices are not even the right size. The reason the above runs > is because nothing you have done with "pc" is being used: it refers to a > "dksp" which you are leaking without using. > > >> >> KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); >> KSPSolve(ksp, breal, xreal_harm); >> KSPSolve(ksp, bimag, ximag_harm); >> ////////////////////////////////////////////////////////// >> >> > On Sun, Jun 10, 2012 at 7:32 PM, Nakib Haider Protik >> > wrote: >> > >> >> Okay. But if I understood correctly, doing the following would also >> be >> >> wrong since I will have made three extra outer KSP objects. I am not >> >> sure >> >> I understand what these functions want. >> >> >> >> KSP cksp, uksp, dksp; >> >> PCMGGetCoarseSolve(pc, &cksp);/**/ >> >> PCMGGetSmootherDown(pc, 0, &dksp);/**/ >> >> PCMGGetSmootherUp(pc, 1, &uksp);/**/ >> >> >> > >> > This is fine, all of these are getting references to inner objects >> without >> > overwriting the outer objects. Of course you don't need to get these >> > references if you aren't going to modify them. >> > >> > I recommend that you experiment with algorithms using the command line >> and >> > check that it does what you expect using -ksp_view. >> > >> >> >> -- >> Nakib :) >> > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 20:22:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 20:22:13 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <54026.99.254.3.78.1339377577.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51749.99.254.3.78.1339361344.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> <54026.99.254.3.78.1339377577.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 8:19 PM, Nakib Haider Protik wrote: > With the suggested changes, the code doesn't run. > You didn't state the error or show the code. > Is there any example > code of KSP solvers using multigrid preconditioning? All the examples of > multigrid I have come across involve dmmg. > Please upgrade to petsc-3.3. DMMG is gone and most examples support multigrid. In most cases, you can just run with -pc_type mg. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Sun Jun 10 20:32:37 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Sun, 10 Jun 2012 21:32:37 -0400 (EDT) Subject: [petsc-users] multigrid implementation problem In-Reply-To: References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> <54026.99.254.3.78.1339377577.squirrel@webmail01.uottawa.ca> Message-ID: <54121.99.254.3.78.1339378357.squirrel@webmail01.uottawa.ca> Here's the code with the changes you suggested followed by the terminal output: KSPCreate(PETSC_COMM_WORLD, &ksp); //KSPCreate(PETSC_COMM_WORLD, &cksp); //KSPCreate(PETSC_COMM_WORLD, &uksp); //KSPCreate(PETSC_COMM_WORLD, &dksp); //KSPSetType(ksp, KSPGMRES); //KSPSetType(cksp, KSPGMRES); //KSPSetType(uksp, KSPGMRES); //KSPSetType(dksp, KSPGMRES); KSPGetPC(ksp, &pc); //KSPGetPC(cksp, &pc); //KSPGetPC(uksp, &pc); //KSPGetPC(dksp, &pc); KSPSetFromOptions(ksp); PCSetType(pc, PCMG); PCMGSetLevels(pc, 2, PETSC_NULL); PCMGSetType(pc, PC_MG_MULTIPLICATIVE); PCMGSetCycleType(pc, PC_MG_CYCLE_V); MatDuplicate(A, MAT_COPY_VALUES, &P); PCMGSetCyclesOnLevel(pc, 0, 1); PCMGSetCyclesOnLevel(pc, 1, 1); //PCMGGetCoarseSolve(pc, &cksp); //PCMGGetSmootherDown(pc, 0, &dksp); //PCMGGetSmootherUp(pc, 1, &uksp); PCMGSetInterpolation(pc, 1, P); PCMGSetRestriction(pc, 1, P); PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); KSPSolve(ksp, breal, xreal_harm); KSPSolve(ksp, bimag, ximag_harm); [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 12, Tue Mar 16 23:20:08 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./3dPoissonSolver-cyl-test on a linux-gnu named nakib-F82Q by nakib Sun Jun 10 21:30:04 2012 [0]PETSC ERROR: Libraries linked from /home/nakib/petsc-3.0.0-p12/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Wed May 9 13:44:03 2012 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 --with-shared=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatGetVecs() line 7098 in src/mat/interface/matrix.c [0]PETSC ERROR: KSPGetVecs() line 820 in src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: PCSetUp_MG() line 465 in src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 12, Tue Mar 16 23:20:08 CDT 2010 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./3dPoissonSolver-cyl-test on a linux-gnu named nakib-F82Q by nakib Sun Jun 10 21:30:04 2012 [0]PETSC ERROR: Libraries linked from /home/nakib/petsc-3.0.0-p12/linux-gnu-c-debug/lib [0]PETSC ERROR: Configure run at Wed May 9 13:44:03 2012 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-mpich=1 --with-shared=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatGetVecs() line 7098 in src/mat/interface/matrix.c [0]PETSC ERROR: KSPGetVecs() line 820 in src/ksp/ksp/interface/iterativ.c [0]PETSC ERROR: PCSetUp_MG() line 465 in src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c The code converges to the wrong solution too. Thanks > On Sun, Jun 10, 2012 at 8:19 PM, Nakib Haider Protik > wrote: > >> With the suggested changes, the code doesn't run. >> > > You didn't state the error or show the code. > > >> Is there any example >> code of KSP solvers using multigrid preconditioning? All the examples of >> multigrid I have come across involve dmmg. >> > > Please upgrade to petsc-3.3. DMMG is gone and most examples support > multigrid. In most cases, you can just run with -pc_type mg. > -- Nakib :) From jedbrown at mcs.anl.gov Sun Jun 10 20:49:12 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 20:49:12 -0500 Subject: [petsc-users] multigrid implementation problem In-Reply-To: <54121.99.254.3.78.1339378357.squirrel@webmail01.uottawa.ca> References: <43248.99.254.3.78.1339305056.squirrel@webmail01.uottawa.ca> <51950.99.254.3.78.1339363206.squirrel@webmail01.uottawa.ca> <52263.99.254.3.78.1339364368.squirrel@webmail01.uottawa.ca> <52413.99.254.3.78.1339365288.squirrel@webmail01.uottawa.ca> <53300.99.254.3.78.1339373144.squirrel@webmail01.uottawa.ca> <53771.99.254.3.78.1339374766.squirrel@webmail01.uottawa.ca> <53844.99.254.3.78.1339376067.squirrel@webmail01.uottawa.ca> <54026.99.254.3.78.1339377577.squirrel@webmail01.uottawa.ca> <54121.99.254.3.78.1339378357.squirrel@webmail01.uottawa.ca> Message-ID: On Sun, Jun 10, 2012 at 8:32 PM, Nakib Haider Protik wrote: > Here's the code with the changes you suggested followed by the terminal > output: > > KSPCreate(PETSC_COMM_WORLD, &ksp); > //KSPCreate(PETSC_COMM_WORLD, &cksp); > //KSPCreate(PETSC_COMM_WORLD, &uksp); > //KSPCreate(PETSC_COMM_WORLD, &dksp); > > //KSPSetType(ksp, KSPGMRES); > //KSPSetType(cksp, KSPGMRES); > //KSPSetType(uksp, KSPGMRES); > //KSPSetType(dksp, KSPGMRES); > > KSPGetPC(ksp, &pc); > //KSPGetPC(cksp, &pc); > //KSPGetPC(uksp, &pc); > //KSPGetPC(dksp, &pc); > > KSPSetFromOptions(ksp); > > PCSetType(pc, PCMG); > PCMGSetLevels(pc, 2, PETSC_NULL); > PCMGSetType(pc, PC_MG_MULTIPLICATIVE); > PCMGSetCycleType(pc, PC_MG_CYCLE_V); > MatDuplicate(A, MAT_COPY_VALUES, &P); > PCMGSetCyclesOnLevel(pc, 0, 1); > PCMGSetCyclesOnLevel(pc, 1, 1); > > //PCMGGetCoarseSolve(pc, &cksp); > //PCMGGetSmootherDown(pc, 0, &dksp); > //PCMGGetSmootherUp(pc, 1, &uksp); > > PCMGSetInterpolation(pc, 1, P); > PCMGSetRestriction(pc, 1, P); > PCMGSetResidual(pc, 0, PCMGDefaultResidual, P); > PCMGSetResidual(pc, 1, PCMGDefaultResidual, P); > ^^ I already explained that these lines are semantically incorrect. You are using the matrix P in different places that do not make sense. You have not set an operator for the smoother and you haven't set a vector. You have to do at least one of those if you want to use the "advanced" PCMG interface. Before trying anything else, install petsc-3.3 and read through this example. http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex45.c.html Compile it and run it with ./ex45 -da_refine 3 -ksp_monitor -pc_type mg -ksp_view Read through the output and understand what algorithm it is using. Run with -help and look at the options starting with "-pc_mg" and "-mg_levels". Change the algorithm using command line parameters. Then, and only then, consider using the low-level PCMG interface directly. > > KSPSetOperators(ksp, A, P, SAME_NONZERO_PATTERN); > KSPSolve(ksp, breal, xreal_harm); > KSPSolve(ksp, bimag, ximag_harm); > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 12, Tue Mar 16 23:20:08 > CDT 2010 > I have asked you several times to upgrade to petsc-3.3. Please do that now, before you write back to this list. > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./3dPoissonSolver-cyl-test on a linux-gnu named nakib-F82Q > by nakib Sun Jun 10 21:30:04 2012 > [0]PETSC ERROR: Libraries linked from > /home/nakib/petsc-3.0.0-p12/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Wed May 9 13:44:03 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack=1 --download-mpich=1 --with-shared=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetVecs() line 7098 in src/mat/interface/matrix.c > [0]PETSC ERROR: KSPGetVecs() line 820 in src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: PCSetUp_MG() line 465 in src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 12, Tue Mar 16 23:20:08 > CDT 2010 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./3dPoissonSolver-cyl-test on a linux-gnu named nakib-F82Q > by nakib Sun Jun 10 21:30:04 2012 > [0]PETSC ERROR: Libraries linked from > /home/nakib/petsc-3.0.0-p12/linux-gnu-c-debug/lib > [0]PETSC ERROR: Configure run at Wed May 9 13:44:03 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack=1 --download-mpich=1 --with-shared=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetVecs() line 7098 in src/mat/interface/matrix.c > [0]PETSC ERROR: KSPGetVecs() line 820 in src/ksp/ksp/interface/iterativ.c > [0]PETSC ERROR: PCSetUp_MG() line 465 in src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: PCSetUp() line 794 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c > > The code converges to the wrong solution too. > The error above is fatal, the solver is not configured correctly. You are supposed to catch the error (you can use the CHKERRQ macro). -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Jun 10 21:10:28 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:10:28 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis Message-ID: Hi all, I configured petsc-3.2-p7 with the following options: ./configure --prefix=/Users/aeronova/Development/local/lib64/petsc/petsc-3.2-p7 --download-mpich=1 --download-blacs=1 --download-parmetis=1 --download-metis=1 --download-scalapack=1 --download-mumps=1 --download-umfpack=1 --with-clanguage=C++ , which threw an error saying ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make on Metis: Could not execute "cd /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": In configure.log, there are some messages related to metis: ================================================================================ TEST configureLibrary from config.packages.metis(/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py:417) TESTING: configureLibrary from config.packages.metis(config/BuildSystem/config/package.py:417) Find an installation and check if it can work with PETSc ================================================================================== Checking for a functional metis Looking for METIS in directory starting with metis Found a copy of METIS in metis-4.0.3 Pushing language C Popping language C Have to rebuild metis, /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3/make.inc != /Users/aeronova/Development/local/src/petsc-3.2-p7/arch-darwin-cxx-debug/conf/metis =============================================================================== Compiling & installing Metis; this may take several minutes =============================================================================== sh: cd /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean Executing: cd /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean sh: (cd Lib ; make clean ) rm -f *.o (cd Programs ; make clean ) rm -f *.o (cd Test ; make clean ) rm -f *.o ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make on Metis: Could not execute "cd /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 && make clean && make library && make minstall && make clean": (cd Lib ; make clean ) rm -f *.o (cd Programs ; make clean ) rm -f *.o (cd Test ; make clean ) rm -f *.o make: *** No rule to make target `library'. Stop. ******************************************************************************* File "./config/configure.py", line 283, in petsc_configure framework.configure(out = sys.stdout) File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/framework.py", line 925, in configure child.configure() File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", line 506, in configure self.executeTest(self.configureLibrary) File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/base.py", line 115, in executeTest ret = apply(test, args,kargs) File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", line 433, in configureLibrary for location, directory, lib, incl in self.generateGuesses(): File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", line 228, in generateGuesses d = self.checkDownload(1) File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", line 313, in checkDownload return self.getInstallDir() File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", line 183, in getInstallDir return os.path.abspath(self.Install()) File "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/packages/metis.py", line 64, in Install raise RuntimeError('Error running make on Metis: '+str(e)) I think the error was raised by missing target 'library,' but I'm not sure how to fix it. I'd appreciate any suggestions. K. Lee. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 10 21:13:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 21:13:43 -0500 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Sun, Jun 10, 2012 at 9:10 PM, Kyunghoon Lee wrote: > Hi all, > > I configured petsc-3.2-p7 with the following options: > > ./configure > --prefix=/Users/aeronova/Development/local/lib64/petsc/petsc-3.2-p7 > --download-mpich=1 --download-blacs=1 --download-parmetis=1 > --download-metis=1 --download-scalapack=1 --download-mumps=1 > --download-umfpack=1 --with-clanguage=C++ > Possibly a stale package file in externalpackages/. But you should upgrade to petsc-3.3 before bothering more with this. It has metis-5.0 which fixes many critical bugs. > > , which threw an error saying > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > Error running make on Metis: Could not execute "cd > /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 > && make clean && make library && make minstall && make clean": > > In configure.log, there are some messages related to metis: > > > ================================================================================ > TEST configureLibrary from > config.packages.metis(/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py:417) > TESTING: configureLibrary from > config.packages.metis(config/BuildSystem/config/package.py:417) > Find an installation and check if it can work with PETSc > > ================================================================================== > Checking for a functional metis > Looking for METIS in directory starting with metis > Found a copy of METIS in metis-4.0.3 > Pushing language C > Popping language C > Have to rebuild metis, > /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3/make.inc > != > /Users/aeronova/Development/local/src/petsc-3.2-p7/arch-darwin-cxx-debug/conf/metis > > =============================================================================== > Compiling & installing Metis; this may take > several minutes > > =============================================================================== > > sh: cd > /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 > && make clean && make library && make minstall && make clean > Executing: cd > /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 > && make clean && make library && make minstall && make clean > sh: (cd Lib ; make clean ) > rm -f *.o > (cd Programs ; make clean ) > rm -f *.o > (cd Test ; make clean ) > rm -f *.o > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > Error running make on Metis: Could not execute "cd > /Users/aeronova/Development/local/src/petsc-3.2-p7/externalpackages/metis-4.0.3 > && make clean && make library && make minstall && make clean": > (cd Lib ; make clean ) > rm -f *.o > (cd Programs ; make clean ) > rm -f *.o > (cd Test ; make clean ) > rm -f *.o > make: *** No rule to make target `library'. Stop. > > ******************************************************************************* > File "./config/configure.py", line 283, in petsc_configure > framework.configure(out = sys.stdout) > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/framework.py", > line 925, in configure > child.configure() > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", > line 506, in configure > self.executeTest(self.configureLibrary) > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/base.py", > line 115, in executeTest > ret = apply(test, args,kargs) > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", > line 433, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", > line 228, in generateGuesses > d = self.checkDownload(1) > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", > line 313, in checkDownload > return self.getInstallDir() > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/package.py", > line 183, in getInstallDir > return os.path.abspath(self.Install()) > File > "/Users/aeronova/Development/local/src/petsc-3.2-p7/config/BuildSystem/config/packages/metis.py", > line 64, in Install > raise RuntimeError('Error running make on Metis: '+str(e)) > > I think the error was raised by missing target 'library,' but I'm not sure > how to fix it. I'd appreciate any suggestions. > > K. Lee. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Jun 10 21:23:23 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:23:23 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 10:13 AM, Jed Brown wrote: > On Sun, Jun 10, 2012 at 9:10 PM, Kyunghoon Lee > wrote: > >> Hi all, >> >> I configured petsc-3.2-p7 with the following options: >> >> ./configure >> --prefix=/Users/aeronova/Development/local/lib64/petsc/petsc-3.2-p7 >> --download-mpich=1 --download-blacs=1 --download-parmetis=1 >> --download-metis=1 --download-scalapack=1 --download-mumps=1 >> --download-umfpack=1 --with-clanguage=C++ >> > > Possibly a stale package file in externalpackages/. > > But you should upgrade to petsc-3.3 before bothering more with this. It > has metis-5.0 which fixes many critical bugs. > Thank you for the reply. I checked the metis directory in externalpackages, but there is no file named "state": CHANGES Graphs/ Lib/ Makefile.in VERSION Doc/ INSTALL METISLib/ Programs/ arch-darwin-cxx-debug/ FILES LICENSE Makefile Test/ make.inc Re petsc-3.3, I need to check its compatibility with libmesh, since I need petsc mainly for libmesh. K. Lee. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Jun 10 21:25:58 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 10 Jun 2012 21:25:58 -0500 (CDT) Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Mon, 11 Jun 2012, Kyunghoon Lee wrote: > On Mon, Jun 11, 2012 at 10:13 AM, Jed Brown wrote: > > > On Sun, Jun 10, 2012 at 9:10 PM, Kyunghoon Lee > > wrote: > > > >> Hi all, > >> > >> I configured petsc-3.2-p7 with the following options: > >> > >> ./configure > >> --prefix=/Users/aeronova/Development/local/lib64/petsc/petsc-3.2-p7 > >> --download-mpich=1 --download-blacs=1 --download-parmetis=1 > >> --download-metis=1 --download-scalapack=1 --download-mumps=1 > >> --download-umfpack=1 --with-clanguage=C++ > >> > > > > Possibly a stale package file in externalpackages/. > > > > But you should upgrade to petsc-3.3 before bothering more with this. It > > has metis-5.0 which fixes many critical bugs. > > > > Thank you for the reply. I checked the metis directory in > externalpackages, but there is no file named "state": > > CHANGES Graphs/ Lib/ > Makefile.in VERSION > Doc/ INSTALL METISLib/ > Programs/ arch-darwin-cxx-debug/ > FILES LICENSE Makefile > Test/ make.inc > > Re petsc-3.3, I need to check its compatibility with libmesh, since I need > petsc mainly for libmesh. you don't need --download-metis for petsc-3.2 [--download-parmetis=1 installs metis for you] [--download-metis=1 is broken in petsc-3.2. Also petsc-3.2 doesn't directly use metis] Satish From jedbrown at mcs.anl.gov Sun Jun 10 21:28:06 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 21:28:06 -0500 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Sun, Jun 10, 2012 at 9:23 PM, Kyunghoon Lee wrote: > Re petsc-3.3, I need to check its compatibility with libmesh, since I need > petsc mainly for libmesh. It works. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Jun 10 21:36:48 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:36:48 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: > you don't need --download-metis for petsc-3.2 [--download-parmetis=1 > installs metis for you] > > [--download-metis=1 is broken in petsc-3.2. Also petsc-3.2 doesn't > directly use metis] > > Satish > I check externalpackages of petsc-3.2-p5, which I configured only with parmetis: $ ls MUMPS_4.10.0/ ParMetis-3.2.0-p1/ SCALAPACK-1.7/ UMFPACK-5.5.1/ blacs-dev/ mpich2-1.4.1p1/ , which has no metis. I guess I cannot get metis just by enabling --download-parmetis=1. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Jun 10 21:39:18 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:39:18 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 10:28 AM, Jed Brown wrote: > On Sun, Jun 10, 2012 at 9:23 PM, Kyunghoon Lee > wrote: > >> Re petsc-3.3, I need to check its compatibility with libmesh, since I >> need petsc mainly for libmesh. > > > It works. > If so, I wonder the latest version of slepc, slepc-3.2-p5, also works with petsc-3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Jun 10 21:39:31 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 10 Jun 2012 21:39:31 -0500 (CDT) Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Mon, 11 Jun 2012, Kyunghoon Lee wrote: > > you don't need --download-metis for petsc-3.2 [--download-parmetis=1 > > installs metis for you] > > > > [--download-metis=1 is broken in petsc-3.2. Also petsc-3.2 doesn't > > directly use metis] > > > > Satish > > > > I check externalpackages of petsc-3.2-p5, which I configured only with > parmetis: > > $ ls > MUMPS_4.10.0/ ParMetis-3.2.0-p1/ SCALAPACK-1.7/ UMFPACK-5.5.1/ > blacs-dev/ mpich2-1.4.1p1/ > > , which has no metis. I guess I cannot get metis just by enabling > --download-parmetis=1. Parmetis includes metis sources - so once its installed with --download-metis - you will have libmetis.a and libparmetis.a Satish From aeronova.mailing at gmail.com Sun Jun 10 21:41:38 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:41:38 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: > Parmetis includes metis sources - so once its installed with > --download-metis - you will have libmetis.a and libparmetis.a > > Satish > Yes, you're right. I checked the installed directory of petsc and found both libmetis.a and libparmetis.a. Thank you for the help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 10 21:43:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 10 Jun 2012 21:43:13 -0500 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Sun, Jun 10, 2012 at 9:39 PM, Kyunghoon Lee wrote: > If so, I wonder the latest version of slepc, slepc-3.2-p5, also works with > petsc-3.3. Use slepc-dev, it works with petsc-3.3. http://www.grycap.upv.es/slepc/download/download.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: From aeronova.mailing at gmail.com Sun Jun 10 21:47:04 2012 From: aeronova.mailing at gmail.com (Kyunghoon Lee) Date: Mon, 11 Jun 2012 10:47:04 +0800 Subject: [petsc-users] compilation error of petsc-3.2-p7 with metis In-Reply-To: References: Message-ID: On Mon, Jun 11, 2012 at 10:43 AM, Jed Brown wrote: > On Sun, Jun 10, 2012 at 9:39 PM, Kyunghoon Lee > wrote: > >> If so, I wonder the latest version of slepc, slepc-3.2-p5, also works >> with petsc-3.3. > > > Use slepc-dev, it works with petsc-3.3. > > http://www.grycap.upv.es/slepc/download/download.htm > Thank you for the link. I really appreciate your help. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetano at email.virginia.edu Mon Jun 11 19:19:45 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Mon, 11 Jun 2012 20:19:45 -0400 Subject: [petsc-users] Data saving / viewer Message-ID: I am experimenting with the possible ways to output and view data (vectors?) created in PETSc. I am creating a DMDA object, and I am using a TS object to evolve the solution in time. What is the recommended approach to save data at several timesteps? I have been struggling with HDF5 so far, but I am open to all possible combinations of outputs / external third party viewers. Gaetano From knepley at gmail.com Mon Jun 11 21:16:16 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 12 Jun 2012 10:16:16 +0800 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On Tue, Jun 12, 2012 at 8:19 AM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > I am experimenting with the possible ways to output and view data > (vectors?) created in PETSc. > I am creating a DMDA object, and I am using a TS object to evolve the > solution in time. What is the recommended approach to save data at > several timesteps? I have been struggling with HDF5 so far, but I am > open to all possible combinations of outputs / external third party > viewers. Unless you have other reasons, I would use the binary viewers. Matt > > Gaetano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Jun 12 01:32:22 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 12 Jun 2012 08:32:22 +0200 Subject: [petsc-users] Nullspace issue Message-ID: <4FD6E276.2070607@tu-dresden.de> I've some trouble to setup the nullspace for my equation correctly. It is an implicit coupled Navier-Stokes / Cahn-Hilliard equation (for two phase flow simulation) in 2D. To simulate an "infinite" domain, the upper and lower boundaries are periodic for all 5 components. The pressure and the two variables from the Cahn-Hilliard equation are null-flux (Neumann) on the other two boundaries, the velocity is set to 0 (Dirichlet) on them. I would expect to have a non-empty nullspace. When using richardson/lu, I see the following behavior: 1 KSP preconditioned resid norm 3.130958085604e+04 true resid norm 1.339173230263e-11 ||r(i)||/||b|| 2.249831332702e-16 2 KSP preconditioned resid norm 5.987054624984e+04 true resid norm 8.900173028027e-12 ||r(i)||/||b|| 1.495242564025e-16 3 KSP preconditioned resid norm 8.159870664335e+04 true resid norm 7.429941832854e-12 ||r(i)||/||b|| 1.248241493927e-16 4 KSP preconditioned resid norm 7.087492912608e+04 true resid norm 7.066574102540e-12 ||r(i)||/||b|| 1.187195164260e-16 5 KSP preconditioned resid norm 1.075156685382e+05 true resid norm 6.963290776604e-12 ||r(i)||/||b|| 1.169843408895e-16 6 KSP preconditioned resid norm 9.260088828115e+04 true resid norm 6.996993371924e-12 ||r(i)||/||b|| 1.175505496012e-16 So, in principle the system is solvable, but the preconditioned norm is wrong. I created a nullspace basis that is 1 on all pressure unknowns and 0 on all of the other four components. I can easily verify that this vector is at least an element of the nullspace, as the norm of the vector resulting from MatMult with the assembled matrix and the nullspace basis vector is in the order of 1e-16. I set the nullspace to the system matrix, and rerun with richardson/lu: 0 KSP preconditioned resid norm 6.025198436485e+09 true resid norm 5.952327229149e+04 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.436455497312e+09 true resid norm 9.795164213312e-09 ||r(i)||/||b|| 1.645602440226e-13 2 KSP preconditioned resid norm 3.365625775916e+09 true resid norm 7.488946142386e-09 ||r(i)||/||b|| 1.258154307396e-13 3 KSP preconditioned resid norm 2.543011385543e+09 true resid norm 5.719959416026e-09 ||r(i)||/||b|| 9.609618550566e-14 4 KSP preconditioned resid norm 4.406639844892e+08 true resid norm 6.417414971975e-09 ||r(i)||/||b|| 1.078135446007e-13 5 KSP preconditioned resid norm 2.741735667402e+09 true resid norm 5.646995693759e-09 ||r(i)||/||b|| 9.487038390809e-14 6 KSP preconditioned resid norm 2.427193596206e+09 true resid norm 1.327430895972e-08 ||r(i)||/||b|| 2.230104033043e-13 7 KSP preconditioned resid norm 1.603502416252e+09 true resid norm 1.338139212677e-08 ||r(i)||/||b|| 2.248094167478e-13 8 KSP preconditioned resid norm 1.646847388237e+09 true resid norm 1.287770565938e-08 ||r(i)||/||b|| 2.163474077218e-13 9 KSP preconditioned resid norm 2.136833312630e+09 true resid norm 1.261183165000e-08 ||r(i)||/||b|| 2.118806840498e-13 10 KSP preconditioned resid norm 1.405267823532e+09 true resid norm 2.499790590170e-08 ||r(i)||/||b|| 4.199686095765e-13 The system at all is not solvable anymore. What could be the problem? Some technical issue I forgot about? Or my be the nullspace be wrong? Thanks for any hint! Thomas From jedbrown at mcs.anl.gov Tue Jun 12 07:31:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 07:31:43 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FD6E276.2070607@tu-dresden.de> References: <4FD6E276.2070607@tu-dresden.de> Message-ID: A known null space doesn't really "fix" LU for a singular problem [1]. You need a preconditioner that is *stable*. Also, it sounds like you may have additional vectors in the null space coming from the Cahn-Hilliard variables. [1] One could create a new matrix with space for Lagrange multipliers enforcing the null space and perform the factorization there. But doing this automatically isn't implemented in the library so you'd have to write it. On Tue, Jun 12, 2012 at 1:32 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > I've some trouble to setup the nullspace for my equation correctly. It is > an implicit coupled Navier-Stokes / Cahn-Hilliard equation (for two phase > flow simulation) in 2D. To simulate an "infinite" domain, the upper and > lower boundaries are periodic for all 5 components. The pressure and the > two variables from the Cahn-Hilliard equation are null-flux (Neumann) on > the other two boundaries, the velocity is set to 0 (Dirichlet) on them. I > would expect to have a non-empty nullspace. When using richardson/lu, I see > the following behavior: > > 1 KSP preconditioned resid norm 3.130958085604e+04 true resid norm > 1.339173230263e-11 ||r(i)||/||b|| 2.249831332702e-16 > 2 KSP preconditioned resid norm 5.987054624984e+04 true resid norm > 8.900173028027e-12 ||r(i)||/||b|| 1.495242564025e-16 > 3 KSP preconditioned resid norm 8.159870664335e+04 true resid norm > 7.429941832854e-12 ||r(i)||/||b|| 1.248241493927e-16 > 4 KSP preconditioned resid norm 7.087492912608e+04 true resid norm > 7.066574102540e-12 ||r(i)||/||b|| 1.187195164260e-16 > 5 KSP preconditioned resid norm 1.075156685382e+05 true resid norm > 6.963290776604e-12 ||r(i)||/||b|| 1.169843408895e-16 > 6 KSP preconditioned resid norm 9.260088828115e+04 true resid norm > 6.996993371924e-12 ||r(i)||/||b|| 1.175505496012e-16 > > So, in principle the system is solvable, but the preconditioned norm is > wrong. I created a nullspace basis that is 1 on all pressure unknowns and 0 > on all of the other four components. I can easily verify that this vector > is at least an element of the nullspace, as the norm of the vector > resulting from MatMult with the assembled matrix and the nullspace basis > vector is in the order of 1e-16. I set the nullspace to the system matrix, > and rerun with richardson/lu: > > 0 KSP preconditioned resid norm 6.025198436485e+09 true resid norm > 5.952327229149e+04 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.436455497312e+09 true resid norm > 9.795164213312e-09 ||r(i)||/||b|| 1.645602440226e-13 > 2 KSP preconditioned resid norm 3.365625775916e+09 true resid norm > 7.488946142386e-09 ||r(i)||/||b|| 1.258154307396e-13 > 3 KSP preconditioned resid norm 2.543011385543e+09 true resid norm > 5.719959416026e-09 ||r(i)||/||b|| 9.609618550566e-14 > 4 KSP preconditioned resid norm 4.406639844892e+08 true resid norm > 6.417414971975e-09 ||r(i)||/||b|| 1.078135446007e-13 > 5 KSP preconditioned resid norm 2.741735667402e+09 true resid norm > 5.646995693759e-09 ||r(i)||/||b|| 9.487038390809e-14 > 6 KSP preconditioned resid norm 2.427193596206e+09 true resid norm > 1.327430895972e-08 ||r(i)||/||b|| 2.230104033043e-13 > 7 KSP preconditioned resid norm 1.603502416252e+09 true resid norm > 1.338139212677e-08 ||r(i)||/||b|| 2.248094167478e-13 > 8 KSP preconditioned resid norm 1.646847388237e+09 true resid norm > 1.287770565938e-08 ||r(i)||/||b|| 2.163474077218e-13 > 9 KSP preconditioned resid norm 2.136833312630e+09 true resid norm > 1.261183165000e-08 ||r(i)||/||b|| 2.118806840498e-13 > 10 KSP preconditioned resid norm 1.405267823532e+09 true resid norm > 2.499790590170e-08 ||r(i)||/||b|| 4.199686095765e-13 > > The system at all is not solvable anymore. What could be the problem? Some > technical issue I forgot about? Or my be the nullspace be wrong? Thanks for > any hint! > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Jun 12 07:56:29 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 12 Jun 2012 14:56:29 +0200 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> Message-ID: <4FD73C7D.3030703@tu-dresden.de> There should be no null space from the Cahn-Hilliard equation. Is there some black-box preconditioner that does not relay on LU factorization at some point? I know that black-box approaches are mostly not efficient, but I would have something I can work with. Thomas Am 12.06.2012 14:31, schrieb Jed Brown: > A known null space doesn't really "fix" LU for a singular problem [1]. > You need a preconditioner that is *stable*. Also, it sounds like you > may have additional vectors in the null space coming from the > Cahn-Hilliard variables. > > [1] One could create a new matrix with space for Lagrange multipliers > enforcing the null space and perform the factorization there. But > doing this automatically isn't implemented in the library so you'd > have to write it. > > On Tue, Jun 12, 2012 at 1:32 AM, Thomas Witkowski > > wrote: > > I've some trouble to setup the nullspace for my equation > correctly. It is an implicit coupled Navier-Stokes / Cahn-Hilliard > equation (for two phase flow simulation) in 2D. To simulate an > "infinite" domain, the upper and lower boundaries are periodic for > all 5 components. The pressure and the two variables from the > Cahn-Hilliard equation are null-flux (Neumann) on the other two > boundaries, the velocity is set to 0 (Dirichlet) on them. I would > expect to have a non-empty nullspace. When using richardson/lu, I > see the following behavior: > > 1 KSP preconditioned resid norm 3.130958085604e+04 true resid > norm 1.339173230263e-11 ||r(i)||/||b|| 2.249831332702e-16 > 2 KSP preconditioned resid norm 5.987054624984e+04 true resid > norm 8.900173028027e-12 ||r(i)||/||b|| 1.495242564025e-16 > 3 KSP preconditioned resid norm 8.159870664335e+04 true resid > norm 7.429941832854e-12 ||r(i)||/||b|| 1.248241493927e-16 > 4 KSP preconditioned resid norm 7.087492912608e+04 true resid > norm 7.066574102540e-12 ||r(i)||/||b|| 1.187195164260e-16 > 5 KSP preconditioned resid norm 1.075156685382e+05 true resid > norm 6.963290776604e-12 ||r(i)||/||b|| 1.169843408895e-16 > 6 KSP preconditioned resid norm 9.260088828115e+04 true resid > norm 6.996993371924e-12 ||r(i)||/||b|| 1.175505496012e-16 > > So, in principle the system is solvable, but the preconditioned > norm is wrong. I created a nullspace basis that is 1 on all > pressure unknowns and 0 on all of the other four components. I can > easily verify that this vector is at least an element of the > nullspace, as the norm of the vector resulting from MatMult with > the assembled matrix and the nullspace basis vector is in the > order of 1e-16. I set the nullspace to the system matrix, and > rerun with richardson/lu: > > 0 KSP preconditioned resid norm 6.025198436485e+09 true resid > norm 5.952327229149e+04 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.436455497312e+09 true resid > norm 9.795164213312e-09 ||r(i)||/||b|| 1.645602440226e-13 > 2 KSP preconditioned resid norm 3.365625775916e+09 true resid > norm 7.488946142386e-09 ||r(i)||/||b|| 1.258154307396e-13 > 3 KSP preconditioned resid norm 2.543011385543e+09 true resid > norm 5.719959416026e-09 ||r(i)||/||b|| 9.609618550566e-14 > 4 KSP preconditioned resid norm 4.406639844892e+08 true resid > norm 6.417414971975e-09 ||r(i)||/||b|| 1.078135446007e-13 > 5 KSP preconditioned resid norm 2.741735667402e+09 true resid > norm 5.646995693759e-09 ||r(i)||/||b|| 9.487038390809e-14 > 6 KSP preconditioned resid norm 2.427193596206e+09 true resid > norm 1.327430895972e-08 ||r(i)||/||b|| 2.230104033043e-13 > 7 KSP preconditioned resid norm 1.603502416252e+09 true resid > norm 1.338139212677e-08 ||r(i)||/||b|| 2.248094167478e-13 > 8 KSP preconditioned resid norm 1.646847388237e+09 true resid > norm 1.287770565938e-08 ||r(i)||/||b|| 2.163474077218e-13 > 9 KSP preconditioned resid norm 2.136833312630e+09 true resid > norm 1.261183165000e-08 ||r(i)||/||b|| 2.118806840498e-13 > 10 KSP preconditioned resid norm 1.405267823532e+09 true resid > norm 2.499790590170e-08 ||r(i)||/||b|| 4.199686095765e-13 > > The system at all is not solvable anymore. What could be the > problem? Some technical issue I forgot about? Or my be the > nullspace be wrong? Thanks for any hint! > > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 08:04:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 08:04:22 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FD73C7D.3030703@tu-dresden.de> References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> Message-ID: On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > There should be no null space from the Cahn-Hilliard equation. You said all those boundary conditions are either Neumann or periodic. I guess it couples to the fluid variables without any null space? > Is there some black-box preconditioner that does not relay on LU > factorization at some point? I know that black-box approaches are mostly > not efficient, but I would have something I can work with. The SVD always works and will tell you about a null space, but of course it's very expensive. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Jun 12 08:19:44 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 12 Jun 2012 15:19:44 +0200 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> Message-ID: <4FD741F0.9080004@tu-dresden.de> Am 12.06.2012 15:04, schrieb Jed Brown: > On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski > > wrote: > > There should be no null space from the Cahn-Hilliard equation. > > > You said all those boundary conditions are either Neumann or periodic. > I guess it couples to the fluid variables without any null space? yes. > > Is there some black-box preconditioner that does not relay on LU > factorization at some point? I know that black-box approaches are > mostly not efficient, but I would have something I can work with. > > > The SVD always works and will tell you about a null space, but of > course it's very expensive. So assume I have a basis for the null space of the system that should be solved. Is there any block-box solver/preconditioner approach that does not make use of (I)LU factorization at any point? Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 08:36:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 08:36:19 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FD741F0.9080004@tu-dresden.de> References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> Message-ID: SVD works. Black box solvers aren't really. You can try sparse approximate inverse, but good luck. ;-D If you specify a "near null space", you can try smoothed aggregation. Or just run domain decomposition, the decomposition generally breaks to null space so that factorization can be used on subdomains. On Jun 12, 2012 8:19 AM, "Thomas Witkowski" wrote: > Am 12.06.2012 15:04, schrieb Jed Brown: > > On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> There should be no null space from the Cahn-Hilliard equation. > > > You said all those boundary conditions are either Neumann or periodic. I > guess it couples to the fluid variables without any null space? > > yes. > > > >> Is there some black-box preconditioner that does not relay on LU >> factorization at some point? I know that black-box approaches are mostly >> not efficient, but I would have something I can work with. > > > The SVD always works and will tell you about a null space, but of course > it's very expensive. > > So assume I have a basis for the null space of the system that should be > solved. Is there any block-box solver/preconditioner approach that does not > make use of (I)LU factorization at any point? > > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 12 08:37:50 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Jun 2012 08:37:50 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FD741F0.9080004@tu-dresden.de> References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> Message-ID: Thomas, Since you are solving a coupled system of equations you should start with PCFIELDSPLIT. This allows you to build a preconditioner by combining solvers for separate components of the PDE. It can even be applied recursively to first separate the Navier-Stokes equations from the Cahn-Hillard and then to separate the parts of the Navier-Stokes. It may take a little thought for how you want the separation done and what solvers to use on what parts, but it is the only way to get an efficient solver for such a coupled system. Barry Jed, WTF are you talking about SVDs and stuff? On Jun 12, 2012, at 8:19 AM, Thomas Witkowski wrote: > Am 12.06.2012 15:04, schrieb Jed Brown: >> On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski wrote: >> There should be no null space from the Cahn-Hilliard equation. >> >> You said all those boundary conditions are either Neumann or periodic. I guess it couples to the fluid variables without any null space? > yes. >> >> Is there some black-box preconditioner that does not relay on LU factorization at some point? I know that black-box approaches are mostly not efficient, but I would have something I can work with. >> >> The SVD always works and will tell you about a null space, but of course it's very expensive. > So assume I have a basis for the null space of the system that should be solved. Is there any block-box solver/preconditioner approach that does not make use of (I)LU factorization at any point? > > Thomas > From gle6b at virginia.edu Tue Jun 12 09:36:55 2012 From: gle6b at virginia.edu (Gaetano Esposito) Date: Tue, 12 Jun 2012 10:36:55 -0400 Subject: [petsc-users] Data saving / viewer Message-ID: I was specifically referring to my difficulties on using the HDF5 format to print multiple timesteps as I cannot use VecView with the same vector multiple times. Say that I advance the solution in time and I would like to print the solution every N timesteps. How do I achieve it using VecView operating on the HDF5 viewer? Thanks, --g From knepley at gmail.com Tue Jun 12 09:56:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 12 Jun 2012 22:56:12 +0800 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On Tue, Jun 12, 2012 at 10:36 PM, Gaetano Esposito wrote: > I was specifically referring to my difficulties on using the HDF5 > format to print multiple timesteps as I cannot use VecView with the > same vector multiple times. Say that I advance the solution in time > and I would like to print the solution every N timesteps. How do I > achieve it using VecView operating on the HDF5 viewer? > This is the intention of: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerHDF5SetTimestep.html#PetscViewerHDF5SetTimestep http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerHDF5IncrementTimestep.html#PetscViewerHDF5IncrementTimestep Matt > Thanks, > --g -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetano at email.virginia.edu Tue Jun 12 10:03:14 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Tue, 12 Jun 2012 11:03:14 -0400 Subject: [petsc-users] Data saving / viewer Message-ID: I was specifically trying to refer to my difficulties on using the HDF5 format to print multiple timesteps as I cannot use VecView with the same vector multiple times. Say that I advance the solution in time and I would like to print the solution every N timesteps. How do I achieve it using VecView operating on the HDF5 viewer? Thanks, --g From behzad.baghapour at gmail.com Tue Jun 12 10:50:58 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Tue, 12 Jun 2012 20:20:58 +0430 Subject: [petsc-users] preconditioning with Matrix-Free SNES Message-ID: Dear all, Since the option "-snes_mf" does not call FormJacobian, then how I should set the required preconditioning matrix in the procedure of matrix-free SNES? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 10:52:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 10:52:38 -0500 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: On Tue, Jun 12, 2012 at 10:50 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Since the option "-snes_mf" does not call FormJacobian, then how I should > set the required preconditioning matrix in the procedure of matrix-free > SNES? Use -snes_mf_operator instead of -snes_mf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 10:54:35 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 10:54:35 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> Message-ID: On Tue, Jun 12, 2012 at 8:37 AM, Barry Smith wrote: > Jed, WTF are you talking about SVDs and stuff? It's "black box" and works, if you are sufficiently unambitious and have endless time to wait. -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Tue Jun 12 11:12:19 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Tue, 12 Jun 2012 20:42:19 +0430 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: Thank you. Now, I received the error which I should change Mat into mffd. On Tue, Jun 12, 2012 at 8:22 PM, Jed Brown wrote: > On Tue, Jun 12, 2012 at 10:50 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> Since the option "-snes_mf" does not call FormJacobian, then how I should >> set the required preconditioning matrix in the procedure of matrix-free >> SNES? > > > Use -snes_mf_operator instead of -snes_mf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 11:17:39 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 11:17:39 -0500 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Thank you. Now, I received the error which I should change Mat into mffd. *Always* include the entire error message! You have to assemble a matrix to use most preconditioners. -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Tue Jun 12 12:46:24 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Tue, 12 Jun 2012 22:16:24 +0430 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: You are right! I do calculate the Jacobian and Preconditioning matrice in FormJacobian function with Mat context. When I used "-snes_mf_operator" I received the error: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Mat type mffd! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue Jun 12 22:01:48 2012 [0]PETSC ERROR: Libraries linked from /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 --download-hypre --download-parms --with-clanguage=cxx -with-debugging=no [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatZeroEntries() line 5186 in src/mat/interface/matrix.c [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue Jun 12 22:01:48 2012 [0]PETSC ERROR: Libraries linked from /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 --download-hypre --download-parms --with-clanguage=cxx -with-debugging=no [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 make: [run] Error 59 (ignored) On Tue, Jun 12, 2012 at 8:47 PM, Jed Brown wrote: > On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> Thank you. Now, I received the error which I should change Mat into mffd. > > > *Always* include the entire error message! > > You have to assemble a matrix to use most preconditioners. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 12 12:54:48 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jun 2012 01:54:48 +0800 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: On Wed, Jun 13, 2012 at 1:46 AM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > You are right! I do calculate the Jacobian and Preconditioning matrice in > FormJacobian function with Mat context. When I used "-snes_mf_operator" I > received the error: > Right, you only want to assemble the preconditioning matrix if you use MF. Matt > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type mffd! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue > Jun 12 22:01:48 2012 > [0]PETSC ERROR: Libraries linked from > /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-f2cblaslapack=1 --download-mpich=1 --download-hypre > --download-parms --with-clanguage=cxx -with-debugging=no > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: MatZeroEntries() line 5186 in src/mat/interface/matrix.c > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue > Jun 12 22:01:48 2012 > [0]PETSC ERROR: Libraries linked from > /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-f2cblaslapack=1 --download-mpich=1 --download-hypre > --download-parms --with-clanguage=cxx -with-debugging=no > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > make: [run] Error 59 (ignored) > > > > On Tue, Jun 12, 2012 at 8:47 PM, Jed Brown wrote: > >> On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> >>> Thank you. Now, I received the error which I should change Mat into mffd. >> >> >> *Always* include the entire error message! >> >> You have to assemble a matrix to use most preconditioners. >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Tue Jun 12 13:05:02 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 12 Jun 2012 20:05:02 +0200 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> Message-ID: <4FD784CE.5030700@tu-dresden.de> From the three available block algorithms of the fieldsplit preconditioner, I could use only block jacobi and gauss seidel ones. For the Schur type approach I have no idea what to do with the Schur complement in the case block one is the Navier Stokes equation and block two is the Cahn Hilliard equation. Is there any assumption (that could also be verified in practice) on the system that these block preconditioners will work? What is your experience? If there are good preconditioners for the two blocks available, how good will pcfieldsplit preconditioner work? Thomas Am 12.06.2012 15:37, schrieb Barry Smith: > Thomas, > > Since you are solving a coupled system of equations you should start with PCFIELDSPLIT. This allows you to build a preconditioner by combining solvers for separate components of the PDE. It can even be applied recursively to first separate the Navier-Stokes equations from the Cahn-Hillard and then to separate the parts of the Navier-Stokes. > > It may take a little thought for how you want the separation done and what solvers to use on what parts, but it is the only way to get an efficient solver for such a coupled system. > > Barry > > Jed, WTF are you talking about SVDs and stuff? > > > On Jun 12, 2012, at 8:19 AM, Thomas Witkowski wrote: > >> Am 12.06.2012 15:04, schrieb Jed Brown: >>> On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski wrote: >>> There should be no null space from the Cahn-Hilliard equation. >>> >>> You said all those boundary conditions are either Neumann or periodic. I guess it couples to the fluid variables without any null space? >> yes. >>> >>> Is there some black-box preconditioner that does not relay on LU factorization at some point? I know that black-box approaches are mostly not efficient, but I would have something I can work with. >>> >>> The SVD always works and will tell you about a null space, but of course it's very expensive. >> So assume I have a basis for the null space of the system that should be solved. Is there any block-box solver/preconditioner approach that does not make use of (I)LU factorization at any point? >> >> Thomas >> From gle6b at virginia.edu Tue Jun 12 12:35:14 2012 From: gle6b at virginia.edu (Gaetano Esposito) Date: Tue, 12 Jun 2012 13:35:14 -0400 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: Thanks, I was aware of such functions. I think I may have found where my confusion is. If I have a regular Vec, HDF5 SetTimestep and IncrementTimestep work as expected. Instead the following does not work for me: [...] DMCreateGlobalVector(da, &solution); [...] // fill the vector solution and restore HDF5Open HDF5SetTimestep(H5_viewer, 0); ierr = VecView(solution,H5_viewer); ERRCHKQ(ierr); HDF5IncrementTimestep(H5_viewer); ierr = VecView(solution,H5_viewer); ERRCHKQ(ierr); [...] I get the following external library error from HDF at the line of the second VecView: HDF5-DIAG: Error detected in HDF5 (1.8.6) MPI-process 0: #000: H5D.c line 170 in H5Dcreate2(): unable to create dataset major: Dataset minor: Unable to initialize object #001: H5Dint.c line 431 in H5D_create_named(): unable to create and link to dataset major: Dataset minor: Unable to initialize object #002: H5L.c line 1640 in H5L_link_object(): unable to create new link to object major: Links minor: Unable to initialize object #003: H5L.c line 1866 in H5L_create_real(): can't insert link major: Symbol table minor: Unable to insert object #004: H5Gtraverse.c line 929 in H5G_traverse(): internal path traversal failed major: Symbol table minor: Object not found #005: H5Gtraverse.c line 718 in H5G_traverse_real(): traversal operator failed major: Symbol table minor: Callback failed #006: H5L.c line 1675 in H5L_link_cb(): name already exists major: Symbol table minor: Object already exists From knepley at gmail.com Tue Jun 12 13:27:59 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jun 2012 02:27:59 +0800 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On Wed, Jun 13, 2012 at 1:35 AM, Gaetano Esposito wrote: > Thanks, I was aware of such functions. I think I may have found where > my confusion is. If I have a regular Vec, HDF5 SetTimestep and > IncrementTimestep work as expected. > Instead the following does not work for me: > Is this with the newest release? Matt > [...] > DMCreateGlobalVector(da, &solution); > [...] // fill the vector solution and restore > HDF5Open > HDF5SetTimestep(H5_viewer, 0); > > ierr = VecView(solution,H5_viewer); ERRCHKQ(ierr); > HDF5IncrementTimestep(H5_viewer); > ierr = VecView(solution,H5_viewer); ERRCHKQ(ierr); > [...] > > I get the following external library error from HDF at the line of the > second VecView: > > HDF5-DIAG: Error detected in HDF5 (1.8.6) MPI-process 0: > #000: H5D.c line 170 in H5Dcreate2(): unable to create dataset > major: Dataset > minor: Unable to initialize object > #001: H5Dint.c line 431 in H5D_create_named(): unable to create and > link to dataset > major: Dataset > minor: Unable to initialize object > #002: H5L.c line 1640 in H5L_link_object(): unable to create new > link to object > major: Links > minor: Unable to initialize object > #003: H5L.c line 1866 in H5L_create_real(): can't insert link > major: Symbol table > minor: Unable to insert object > #004: H5Gtraverse.c line 929 in H5G_traverse(): internal path traversal > failed > major: Symbol table > minor: Object not found > #005: H5Gtraverse.c line 718 in H5G_traverse_real(): traversal operator > failed > major: Symbol table > minor: Callback failed > #006: H5L.c line 1675 in H5L_link_cb(): name already exists > major: Symbol table > minor: Object already exists > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetano at email.virginia.edu Tue Jun 12 13:33:30 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Tue, 12 Jun 2012 14:33:30 -0400 Subject: [petsc-users] Data saving / viewer Message-ID: It is petsc-3.2-p7. From knepley at gmail.com Tue Jun 12 13:52:56 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jun 2012 02:52:56 +0800 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On Wed, Jun 13, 2012 at 2:33 AM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > It is petsc-3.2-p7. > Try it with 3.3 Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 12 15:35:13 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Jun 2012 15:35:13 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FD784CE.5030700@tu-dresden.de> References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> <4FD784CE.5030700@tu-dresden.de> Message-ID: <4FB5731E-9BD1-4389-BDCA-6B8F508B160A@mcs.anl.gov> On Jun 12, 2012, at 1:05 PM, Thomas Witkowski wrote: > From the three available block algorithms of the fieldsplit preconditioner, I could use only block jacobi and gauss seidel ones. For the Schur type approach I have no idea what to do with the Schur complement in the case block one is the Navier Stokes equation and block two is the Cahn Hilliard equation. Is there any assumption (that could also be verified in practice) on the system that these block preconditioners will work? What is your experience? If there are good preconditioners for the two blocks available, how good will pcfieldsplit preconditioner work? > G-S and Shur complement type fieldsplit PCs almost always work right so long as you have decent preconditioners for the original block(s). Often you don't need much of anything to precondition the Schur complement. As I said before you can construct a preconditioner for the N.S. block by again using a PCFIELDSPLIT on that (recursive use of PCs). Note that once you have defined your blocks (in the simple case if you have all your degrees of freedom collocated (no staggered grid) just set the Vec and Mat block size; otherwise you need to construct an IS that defines each block) then experimenting is easy and can be done at the command line without changing the code. Barry > Thomas > > Am 12.06.2012 15:37, schrieb Barry Smith: >> Thomas, >> >> Since you are solving a coupled system of equations you should start with PCFIELDSPLIT. This allows you to build a preconditioner by combining solvers for separate components of the PDE. It can even be applied recursively to first separate the Navier-Stokes equations from the Cahn-Hillard and then to separate the parts of the Navier-Stokes. >> >> It may take a little thought for how you want the separation done and what solvers to use on what parts, but it is the only way to get an efficient solver for such a coupled system. >> >> Barry >> >> Jed, WTF are you talking about SVDs and stuff? >> >> >> On Jun 12, 2012, at 8:19 AM, Thomas Witkowski wrote: >> >>> Am 12.06.2012 15:04, schrieb Jed Brown: >>>> On Tue, Jun 12, 2012 at 7:56 AM, Thomas Witkowski wrote: >>>> There should be no null space from the Cahn-Hilliard equation. >>>> >>>> You said all those boundary conditions are either Neumann or periodic. I guess it couples to the fluid variables without any null space? >>> yes. >>>> >>>> Is there some black-box preconditioner that does not relay on LU factorization at some point? I know that black-box approaches are mostly not efficient, but I would have something I can work with. >>>> >>>> The SVD always works and will tell you about a null space, but of course it's very expensive. >>> So assume I have a basis for the null space of the system that should be solved. Is there any block-box solver/preconditioner approach that does not make use of (I)LU factorization at any point? >>> >>> Thomas >>> > From bsmith at mcs.anl.gov Tue Jun 12 15:39:19 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Jun 2012 15:39:19 -0500 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: Message-ID: <908B7CB4-4583-41E1-A29B-71E6262027B8@mcs.anl.gov> Like in your ComputeJacobian( Mat *J,Mat *B, ...) you are calling MatZeroEntries(*J....) and then MatSetValues(*J,....) you should not do that. You should call the zero entries and set values on B only. Then call MatAssemblyBegin/End() separately on BOTH matrices. Also if you checked the error code from MatZeroEntries() with a CHKERRQ(ierr) instead of ignoring it you would likely not get the crash below but would get stack frames instead. Barry On Jun 12, 2012, at 12:54 PM, Matthew Knepley wrote: > On Wed, Jun 13, 2012 at 1:46 AM, behzad baghapour wrote: > You are right! I do calculate the Jacobian and Preconditioning matrice in FormJacobian function with Mat context. When I used "-snes_mf_operator" I received the error: > > Right, you only want to assemble the preconditioning matrix if you use MF. > > Matt > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Mat type mffd! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue Jun 12 22:01:48 2012 > [0]PETSC ERROR: Libraries linked from /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 --download-hypre --download-parms --with-clanguage=cxx -with-debugging=no > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatZeroEntries() line 5186 in src/mat/interface/matrix.c > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue Jun 12 22:01:48 2012 > [0]PETSC ERROR: Libraries linked from /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 --download-hypre --download-parms --with-clanguage=cxx -with-debugging=no > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > make: [run] Error 59 (ignored) > > > > On Tue, Jun 12, 2012 at 8:47 PM, Jed Brown wrote: > On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour wrote: > Thank you. Now, I received the error which I should change Mat into mffd. > > Always include the entire error message! > > You have to assemble a matrix to use most preconditioners. > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From jedbrown at mcs.anl.gov Tue Jun 12 15:42:20 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 15:42:20 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: <4FB5731E-9BD1-4389-BDCA-6B8F508B160A@mcs.anl.gov> References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> <4FD784CE.5030700@tu-dresden.de> <4FB5731E-9BD1-4389-BDCA-6B8F508B160A@mcs.anl.gov> Message-ID: On Tue, Jun 12, 2012 at 3:35 PM, Barry Smith wrote: > G-S and Shur complement type fieldsplit PCs almost always work right so > long as you have decent preconditioners for the original block(s). Often > you don't need much of anything to precondition the Schur complement. > Thomas, note that you'll need Schur or some surrogate preconditioner to deal with incompressibility. It'll be worth understanding fieldsplit variants while working with Navier-Stokes and Cahn-Hilliard separately. Once you understand how to "drive" fieldsplit on those systems separately, you can put them together. > > As I said before you can construct a preconditioner for the N.S. block > by again using a PCFIELDSPLIT on that (recursive use of PCs). > > Note that once you have defined your blocks (in the simple case if you > have all your degrees of freedom collocated (no staggered grid) just set > the Vec and Mat block size; otherwise you need to construct an IS that > defines each block) then experimenting is easy and can be done at the > command line without changing the code. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 12 15:47:28 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 12 Jun 2012 15:47:28 -0500 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> <4FD784CE.5030700@tu-dresden.de> <4FB5731E-9BD1-4389-BDCA-6B8F508B160A@mcs.anl.gov> Message-ID: On Jun 12, 2012, at 3:42 PM, Jed Brown wrote: > On Tue, Jun 12, 2012 at 3:35 PM, Barry Smith wrote: > G-S and Shur complement type fieldsplit PCs almost always work right so long as you have decent preconditioners for the original block(s). Often you don't need much of anything to precondition the Schur complement. > > Thomas, note that you'll need Schur or some surrogate preconditioner to deal with incompressibility. It'll be worth understanding fieldsplit variants while working with Navier-Stokes and Cahn-Hilliard separately. Once you understand how to "drive" fieldsplit on those systems separately, you can put them together. Very good point Jed. Until you can solve N.S. efficiently (for example using PCFIELDSPLIT etc) you really cannot put them together. But that exercise is not "wasted" effort because you will understand fieldsplit and Schur preconditioners and then extending to C.H will be much easier. PCFIELDSPLIT is very much a divide and conquer algorithmic approach, for for divide and conquer to work you need to know how to conquer the pieces. Barry > > > As I said before you can construct a preconditioner for the N.S. block by again using a PCFIELDSPLIT on that (recursive use of PCs). > > Note that once you have defined your blocks (in the simple case if you have all your degrees of freedom collocated (no staggered grid) just set the Vec and Mat block size; otherwise you need to construct an IS that defines each block) then experimenting is easy and can be done at the command line without changing the code. > From gaetano at email.virginia.edu Tue Jun 12 17:27:28 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Tue, 12 Jun 2012 18:27:28 -0400 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On petsc-3.3, using the same code that I reported earlier, I get the following error: HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0: #000: H5D.c line 1102 in H5Dset_extent(): unable to set extend dataset major: Dataset minor: Unable to initialize object #001: H5Dint.c line 2152 in H5D_set_extent(): dataset has contiguous storage major: Invalid arguments to routine minor: Out of range [0]PETSC ERROR: VecView_MPI_HDF5_DA() line 330 in src/dm/impls/da/gr2.c [0]PETSC ERROR: VecView_MPI_DA() line 486 in src/dm/impls/da/gr2.c [0]PETSC ERROR: VecView() line 776 in src/vec/vec/interface/vector.c [0]PETSC ERROR: main() line 137 in /archive03/test_petsc/c++laplace.cpp application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0 From knepley at gmail.com Tue Jun 12 17:50:54 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jun 2012 06:50:54 +0800 Subject: [petsc-users] Data saving / viewer In-Reply-To: References: Message-ID: On Wed, Jun 13, 2012 at 6:27 AM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > On petsc-3.3, using the same code that I reported earlier, I get the > following error: > > HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0: > #000: H5D.c line 1102 in H5Dset_extent(): unable to set extend dataset > major: Dataset > minor: Unable to initialize object > #001: H5Dint.c line 2152 in H5D_set_extent(): dataset has contiguous > storage > major: Invalid arguments to routine > minor: Out of range > [0]PETSC ERROR: VecView_MPI_HDF5_DA() line 330 in src/dm/impls/da/gr2.c > [0]PETSC ERROR: VecView_MPI_DA() line 486 in src/dm/impls/da/gr2.c > [0]PETSC ERROR: VecView() line 776 in src/vec/vec/interface/vector.c > [0]PETSC ERROR: main() line 137 in /archive03/test_petsc/c++laplace.cpp > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0 > Now it depends on exactly what you are doing here, since the error is from HDF5 which thinks that something is inconsistent. Please send a small example that reproduces this error to petsc-maint at mcs.anl.gov. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Tue Jun 12 23:32:03 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Wed, 13 Jun 2012 09:02:03 +0430 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: <908B7CB4-4583-41E1-A29B-71E6262027B8@mcs.anl.gov> References: <908B7CB4-4583-41E1-A29B-71E6262027B8@mcs.anl.gov> Message-ID: Thanks. I did above procedure to my ComputeJacobian (only set values for precondition matrix B and assemble both A and B). This tackles the problem but the linear solver did not converged and bring the below error: 0: CFL = 100, Nonlinear = 0.0626958, Linear = (0,0,1e-05) 3.63577 ----------------------------------------------------------------------------- Linear solve did not converge due to DIVERGED_ITS iterations 30 1: CFL = 100, Nonlinear = 0.108225, Linear = (30,0.047985,0.3) 0.0569319 ----------------------------------------------------------------------------- [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Infinite or not-a-number generated in norm! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./code on a linux-gnu named baghapour by baghapour Wed Jun 13 08:54:57 2012 [0]PETSC ERROR: Libraries linked from /home/baghapour/softs/petsc/linux-gnu-cxx-debug/lib [0]PETSC ERROR: Configure run at Wed Nov 9 19:16:47 2011 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 --with-clanguage=cxx --with-debugging=no --download-parms --download-hypre [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: MatMFFDCompute_WP() line 75 in src/mat/impls/mffd/wp.c [0]PETSC ERROR: MatMult_MFFD() line 351 in src/mat/impls/mffd/mffd.c [0]PETSC ERROR: MatMult() line 2177 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 610 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 156 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 3396 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 190 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c [0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 141 in Newton.cpp Does it need to set specific options for linear solver (KSP) dealing with MF? On Wed, Jun 13, 2012 at 1:09 AM, Barry Smith wrote: > > Like in your ComputeJacobian( Mat *J,Mat *B, ...) > > you are calling MatZeroEntries(*J....) and then MatSetValues(*J,....) > > you should not do that. You should call the zero entries and set values > on B only. Then call MatAssemblyBegin/End() separately on BOTH matrices. > > Also if you checked the error code from MatZeroEntries() with a > CHKERRQ(ierr) instead of ignoring it you would likely not get the crash > below but would get stack frames instead. > > Barry > > On Jun 12, 2012, at 12:54 PM, Matthew Knepley wrote: > > > On Wed, Jun 13, 2012 at 1:46 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > > You are right! I do calculate the Jacobian and Preconditioning matrice > in FormJacobian function with Mat context. When I used "-snes_mf_operator" > I received the error: > > > > Right, you only want to assemble the preconditioning matrix if you use > MF. > > > > Matt > > > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: No support for this operation for this object type! > > [0]PETSC ERROR: Mat type mffd! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 > 10:28:33 CDT 2011 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue > Jun 12 22:01:48 2012 > > [0]PETSC ERROR: Libraries linked from > /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-f2cblaslapack=1 --download-mpich=1 --download-hypre > --download-parms --with-clanguage=cxx -with-debugging=no > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: MatZeroEntries() line 5186 in src/mat/interface/matrix.c > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try > http://valgrind.org on GNU/linux and Apple Mac OS X to find memory > corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > > [0]PETSC ERROR: Signal received! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 > 10:28:33 CDT 2011 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad Tue > Jun 12 22:01:48 2012 > > [0]PETSC ERROR: Libraries linked from > /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib > > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-f2cblaslapack=1 --download-mpich=1 --download-hypre > --download-parms --with-clanguage=cxx -with-debugging=no > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > [cli_0]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > make: [run] Error 59 (ignored) > > > > > > > > On Tue, Jun 12, 2012 at 8:47 PM, Jed Brown wrote: > > On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > > Thank you. Now, I received the error which I should change Mat into mffd. > > > > Always include the entire error message! > > > > You have to assemble a matrix to use most preconditioners. > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 12 23:36:34 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 12 Jun 2012 23:36:34 -0500 Subject: [petsc-users] preconditioning with Matrix-Free SNES In-Reply-To: References: <908B7CB4-4583-41E1-A29B-71E6262027B8@mcs.anl.gov> Message-ID: Run in a debugger with -fp_trap to see where the bad value is generated. Use SNESSetFunctionDomainError() if appropriate. On Tue, Jun 12, 2012 at 11:32 PM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Thanks. I did above procedure to my ComputeJacobian (only set values for > precondition matrix B and assemble both A and B). This tackles the problem > but the linear solver did not converged and bring the below error: > > 0: CFL = 100, Nonlinear = 0.0626958, Linear = (0,0,1e-05) 3.63577 > > ----------------------------------------------------------------------------- > Linear solve did not converge due to DIVERGED_ITS iterations 30 > 1: CFL = 100, Nonlinear = 0.108225, Linear = (30,0.047985,0.3) 0.0569319 > > ----------------------------------------------------------------------------- > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 > CDT 2011 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./code on a linux-gnu named baghapour by baghapour Wed Jun > 13 08:54:57 2012 > [0]PETSC ERROR: Libraries linked from > /home/baghapour/softs/petsc/linux-gnu-cxx-debug/lib > [0]PETSC ERROR: Configure run at Wed Nov 9 19:16:47 2011 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1 > --with-clanguage=cxx --with-debugging=no --download-parms --download-hypre > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c > [0]PETSC ERROR: MatMFFDCompute_WP() line 75 in src/mat/impls/mffd/wp.c > [0]PETSC ERROR: MatMult_MFFD() line 351 in src/mat/impls/mffd/mffd.c > [0]PETSC ERROR: MatMult() line 2177 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 610 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 156 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in > src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 3396 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 190 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c > [0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 141 in Newton.cpp > > Does it need to set specific options for linear solver (KSP) dealing with > MF? > > > > > > > > > On Wed, Jun 13, 2012 at 1:09 AM, Barry Smith wrote: > >> >> Like in your ComputeJacobian( Mat *J,Mat *B, ...) >> >> you are calling MatZeroEntries(*J....) and then MatSetValues(*J,....) >> >> you should not do that. You should call the zero entries and set >> values on B only. Then call MatAssemblyBegin/End() separately on BOTH >> matrices. >> >> Also if you checked the error code from MatZeroEntries() with a >> CHKERRQ(ierr) instead of ignoring it you would likely not get the crash >> below but would get stack frames instead. >> >> Barry >> >> On Jun 12, 2012, at 12:54 PM, Matthew Knepley wrote: >> >> > On Wed, Jun 13, 2012 at 1:46 AM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> > You are right! I do calculate the Jacobian and Preconditioning matrice >> in FormJacobian function with Mat context. When I used "-snes_mf_operator" >> I received the error: >> > >> > Right, you only want to assemble the preconditioning matrix if you use >> MF. >> > >> > Matt >> > >> > [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > [0]PETSC ERROR: No support for this operation for this object type! >> > [0]PETSC ERROR: Mat type mffd! >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 >> 10:28:33 CDT 2011 >> > [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> > [0]PETSC ERROR: See docs/index.html for manual pages. >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad >> Tue Jun 12 22:01:48 2012 >> > [0]PETSC ERROR: Libraries linked from >> /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib >> > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 >> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-f2cblaslapack=1 --download-mpich=1 --download-hypre >> --download-parms --with-clanguage=cxx -with-debugging=no >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: MatZeroEntries() line 5186 in src/mat/interface/matrix.c >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > [0]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [0]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR: or try >> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, >> and run >> > [0]PETSC ERROR: to get more information on the crash. >> > [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> > [0]PETSC ERROR: Signal received! >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 >> 10:28:33 CDT 2011 >> > [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> > [0]PETSC ERROR: See docs/index.html for manual pages. >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: ./code on a linux-gnu named behzad-desktop by behzad >> Tue Jun 12 22:01:48 2012 >> > [0]PETSC ERROR: Libraries linked from >> /home/behzad/softs/petsc/linux-gnu-cxx-debug/lib >> > [0]PETSC ERROR: Configure run at Sat Dec 17 10:21:01 2011 >> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-f2cblaslapack=1 --download-mpich=1 --download-hypre >> --download-parms --with-clanguage=cxx -with-debugging=no >> > [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> > [0]PETSC ERROR: User provided function() line 0 in unknown directory >> unknown file >> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> > [cli_0]: aborting job: >> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 >> > make: [run] Error 59 (ignored) >> > >> > >> > >> > On Tue, Jun 12, 2012 at 8:47 PM, Jed Brown >> wrote: >> > On Tue, Jun 12, 2012 at 11:12 AM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> > Thank you. Now, I received the error which I should change Mat into >> mffd. >> > >> > Always include the entire error message! >> > >> > You have to assemble a matrix to use most preconditioners. >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Jun 13 01:13:57 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 13 Jun 2012 08:13:57 +0200 Subject: [petsc-users] Nullspace issue In-Reply-To: References: <4FD6E276.2070607@tu-dresden.de> <4FD73C7D.3030703@tu-dresden.de> <4FD741F0.9080004@tu-dresden.de> <4FD784CE.5030700@tu-dresden.de> <4FB5731E-9BD1-4389-BDCA-6B8F508B160A@mcs.anl.gov> Message-ID: <4FD82FA5.2020904@tu-dresden.de> Barry and Jed: Thank you very much for all the hints. Now I have a clue where I have to start ... Thomas Am 12.06.2012 22:47, schrieb Barry Smith: > On Jun 12, 2012, at 3:42 PM, Jed Brown wrote: > >> On Tue, Jun 12, 2012 at 3:35 PM, Barry Smith wrote: >> G-S and Shur complement type fieldsplit PCs almost always work right so long as you have decent preconditioners for the original block(s). Often you don't need much of anything to precondition the Schur complement. >> >> Thomas, note that you'll need Schur or some surrogate preconditioner to deal with incompressibility. It'll be worth understanding fieldsplit variants while working with Navier-Stokes and Cahn-Hilliard separately. Once you understand how to "drive" fieldsplit on those systems separately, you can put them together. > Very good point Jed. Until you can solve N.S. efficiently (for example using PCFIELDSPLIT etc) you really cannot put them together. But that exercise is not "wasted" effort because you will understand fieldsplit and Schur preconditioners and then extending to C.H will be much easier. PCFIELDSPLIT is very much a divide and conquer algorithmic approach, for for divide and conquer to work you need to know how to conquer the pieces. > > Barry > >> >> >> As I said before you can construct a preconditioner for the N.S. block by again using a PCFIELDSPLIT on that (recursive use of PCs). >> >> Note that once you have defined your blocks (in the simple case if you have all your degrees of freedom collocated (no staggered grid) just set the Vec and Mat block size; otherwise you need to construct an IS that defines each block) then experimenting is easy and can be done at the command line without changing the code. >> From agrayver at gfz-potsdam.de Wed Jun 13 05:45:10 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 13 Jun 2012 12:45:10 +0200 Subject: [petsc-users] Computation to communication ratio for PETSc's routines Message-ID: <4FD86F36.2030705@gfz-potsdam.de> Hello, As stated, I would like to estimate computation to communication ratio for PETSc's linear algebra routines (e.g. MatMult{Transpose}, MatPtAP, MatMatMult etc.) As far as I understand the ratio depends on particular implementation and number of processes one runs application on. So I guess these formulas should be known? If not then I see one way to estimate it. Write app with those operations, parse -log_summary and then divide flops/messages, however the question is what information from -log_summary output should be used for that? Thanks. -- Regards, Alexander From knepley at gmail.com Wed Jun 13 06:24:24 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 13 Jun 2012 19:24:24 +0800 Subject: [petsc-users] Computation to communication ratio for PETSc's routines In-Reply-To: <4FD86F36.2030705@gfz-potsdam.de> References: <4FD86F36.2030705@gfz-potsdam.de> Message-ID: On Wed, Jun 13, 2012 at 6:45 PM, Alexander Grayver wrote: > Hello, > > As stated, I would like to estimate computation to communication ratio for > PETSc's linear algebra routines (e.g. MatMult{Transpose}, MatPtAP, > MatMatMult etc.) > As far as I understand the ratio depends on particular implementation and > number of processes one runs application on. > So I guess these formulas should be known? > It depends on the input data. For example, block diagonal matrices do not communicate in MatMult(). > If not then I see one way to estimate it. Write app with those operations, > parse -log_summary and then divide flops/messages, however the question is > what information from -log_summary output should be used for that? > We give the number of messages and the average message length, so you can get the total length across processes. Since each side records, we divide by 2 (I think, you should check PetscLogView for specifics). You can also use PetscLogViewPython() to simplify parsing. Matt > Thanks. > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Jun 13 06:32:39 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 13 Jun 2012 13:32:39 +0200 Subject: [petsc-users] Search the mailing list does not work Message-ID: <4FD87A57.5050305@tu-dresden.de> My most important knowledge database does not work anymore :) Searching in http://lists.mcs.anl.gov/pipermail/petsc-users/ and http://lists.mcs.anl.gov/pipermail/petsc-dev fails. Has anybody changed something on the configuration of the mailing list? Thomas From jedbrown at mcs.anl.gov Wed Jun 13 06:39:45 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 13 Jun 2012 06:39:45 -0500 Subject: [petsc-users] Search the mailing list does not work In-Reply-To: <4FD87A57.5050305@tu-dresden.de> References: <4FD87A57.5050305@tu-dresden.de> Message-ID: Thanks, we're looking into it. On Wed, Jun 13, 2012 at 6:32 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > My most important knowledge database does not work anymore :) Searching in > http://lists.mcs.anl.gov/**pipermail/petsc-users/and > http://lists.mcs.anl.gov/**pipermail/petsc-devfails. Has anybody changed something on the configuration of the mailing > list? > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Wed Jun 13 07:05:46 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 13 Jun 2012 15:05:46 +0300 Subject: [petsc-users] Search the mailing list does not work In-Reply-To: References: <4FD87A57.5050305@tu-dresden.de> Message-ID: In the meanwhile, use Google's index: query: "site:lists.mcs.anl.gov/pipermail/petsc-users XXX" query: "site:lists.mcs.anl.gov/pipermail/petsc-dev XXX" A On Wed, Jun 13, 2012 at 2:39 PM, Jed Brown wrote: > Thanks, we're looking into it. > > > On Wed, Jun 13, 2012 at 6:32 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> My most important knowledge database does not work anymore :) Searching >> in http://lists.mcs.anl.gov/**pipermail/petsc-users/and >> http://lists.mcs.anl.gov/**pipermail/petsc-devfails. Has anybody changed something on the configuration of the mailing >> list? >> >> Thomas >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 13 08:17:54 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 13 Jun 2012 15:17:54 +0200 Subject: [petsc-users] Computation to communication ratio for PETSc's routines In-Reply-To: References: <4FD86F36.2030705@gfz-potsdam.de> Message-ID: <4FD89302.2070708@gfz-potsdam.de> On 13.06.2012 13:24, Matthew Knepley wrote: > On Wed, Jun 13, 2012 at 6:45 PM, Alexander Grayver > > wrote: > > Hello, > > As stated, I would like to estimate computation to communication > ratio for PETSc's linear algebra routines (e.g. > MatMult{Transpose}, MatPtAP, MatMatMult etc.) > As far as I understand the ratio depends on particular > implementation and number of processes one runs application on. > So I guess these formulas should be known? > > > It depends on the input data. For example, block diagonal matrices do > not communicate in MatMult(). > > If not then I see one way to estimate it. Write app with those > operations, parse -log_summary and then divide flops/messages, > however the question is what information from -log_summary output > should be used for that? > > > We give the number of messages and the average message length, so you > can get the total length across processes. Since > each side records, we divide by 2 (I think, you should check > PetscLogView for specifics). You can also use PetscLogViewPython() > to simplify parsing. Is there total amount of flops given for different operations separately (I see max flops over all processes only)? > > Matt > > Thanks. > > -- > Regards, > Alexander > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Wed Jun 13 10:54:14 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Wed, 13 Jun 2012 16:54:14 +0100 (BST) Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> Message-ID: <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> Thanks for reply! Well at first look, it looks like that M can be put as precondittioning operator for smoother where keeping ksp as preonly in pre-smoother context.? I am not sure, this leads to the preconditioner desired, coz putting M as preconditioner in pre-smoother alters the residual passed to coarse-grid correction,? i.e. CGC corrects the pre-smoothed solution. >________________________________ > From: Jed Brown >To: Abdul Hanan Sheikh ; PETSc users list >Cc: Abdul - CITG >Sent: Friday, 8 June 2012, 19:01 >Subject: Re: [petsc-users] How to implement projection preconditioner? > > >Isn't this just putting M and M_H as the preconditioning operator for the smoother? > > >On Fri, Jun 8, 2012 at 11:58 AM, Abdul Hanan Sheikh wrote: > >Dear all, >> >>Summer greetings, >> >>I am back with an other query. >>Before I successfully implement the projection preconditioner which was simply >> >>the coarse grid correction operator P = I - A*(P*A_H*R); >> >>I implemented this simply keeping both pre and post smoothing dummy in PCMG setup. >> >>Now I need to revise this and re-implement this where I replace A and coarse operator A_H with >> >>preconditioned one i.e. M^-1 A and M^-1 A_H respectively. Thus new projection reads as >> >> >> >> >>P_new = I - (M^-1 A) {P*(M_H^-1 A_H)*R} >> >> >>Any suggestion to implement this in Petsc would be appreciated. >> >> >> >>Thanking in anticipation. >> >>with regards, Abdul >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Jun 13 11:39:55 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 13 Jun 2012 11:39:55 -0500 Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> Message-ID: On Wed, Jun 13, 2012 at 10:54 AM, Abdul Hanan Sheikh wrote: > Thanks for reply! > Well at first look, it looks like that M can be put as precondittioning > operator for smoother > where keeping ksp as preonly in pre-smoother context. > I am not sure, this leads to the preconditioner desired, coz putting M as > preconditioner in > pre-smoother alters the residual passed to coarse-grid correction, i.e. > CGC corrects the pre-smoothed solution. > That's kind of the point of pre-smoothing. Can you explain more precisely (or just try it). > > > > ------------------------------ > *From:* Jed Brown > *To:* Abdul Hanan Sheikh ; PETSc users list < > petsc-users at mcs.anl.gov> > *Cc:* Abdul - CITG > *Sent:* Friday, 8 June 2012, 19:01 > *Subject:* Re: [petsc-users] How to implement projection preconditioner? > > Isn't this just putting M and M_H as the preconditioning operator for the > smoother? > > On Fri, Jun 8, 2012 at 11:58 AM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > Dear all, > Summer greetings, > I am back with an other query. > Before I successfully implement the projection preconditioner which was > simply > the coarse grid correction operator P = I - A*(P*A_H*R); > I implemented this simply keeping both pre and post smoothing dummy in > PCMG setup. > Now I need to revise this and re-implement this where I replace A and > coarse operator A_H with > preconditioned one i.e. M^-1 A and M^-1 A_H respectively. Thus new > projection reads as > > > P_new = I - (M^-1 A) {P*(M_H^-1 A_H)*R} > > Any suggestion to implement this in Petsc would be appreciated. > > Thanking in anticipation. > with regards, Abdul > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 13 14:09:38 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 13 Jun 2012 14:09:38 -0500 Subject: [petsc-users] Computation to communication ratio for PETSc's routines In-Reply-To: <4FD89302.2070708@gfz-potsdam.de> References: <4FD86F36.2030705@gfz-potsdam.de> <4FD89302.2070708@gfz-potsdam.de> Message-ID: On Jun 13, 2012, at 8:17 AM, Alexander Grayver wrote: > On 13.06.2012 13:24, Matthew Knepley wrote: >> On Wed, Jun 13, 2012 at 6:45 PM, Alexander Grayver wrote: >> Hello, >> >> As stated, I would like to estimate computation to communication ratio for PETSc's linear algebra routines (e.g. MatMult{Transpose}, MatPtAP, MatMatMult etc.) >> As far as I understand the ratio depends on particular implementation and number of processes one runs application on. >> So I guess these formulas should be known? >> >> It depends on the input data. For example, block diagonal matrices do not communicate in MatMult(). >> >> If not then I see one way to estimate it. Write app with those operations, parse -log_summary and then divide flops/messages, however the question is what information from -log_summary output should be used for that? >> >> We give the number of messages and the average message length, so you can get the total length across processes. Since >> each side records, we divide by 2 (I think, you should check PetscLogView for specifics). You can also use PetscLogViewPython() >> to simplify parsing. > > Is there total amount of flops given for different operations separately (I see max flops over all processes only)? Sadly no. My original plan was that people who wanted different summary information would copy PetscLogView() and modify it to output just what they want. Sadly that may be too difficult. Barry > >> >> Matt >> >> Thanks. >> >> -- >> Regards, >> Alexander >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > > -- > Regards, > Alexander > From irving at naml.us Wed Jun 13 19:55:30 2012 From: irving at naml.us (Geoffrey Irving) Date: Wed, 13 Jun 2012 17:55:30 -0700 Subject: [petsc-users] MatSetBlockSize no longer works after MatCreateSeqAIJ Message-ID: Hello, The following code used to work, but breaks after upgrading to 3.3: .... CHECK(MatCreateSeqAIJ(comm,3*n,3*n,0,scalar_view(nnz).data(),&mat)); CHECK(MatSetBlockSize(mat,3)); ... The error follows. Presumably MatSetBlockSize got more strict, and I need to break the single MatCreateSeqAIJ into multiple lines (matrix creation followed by preallocation) in order to insert MatSetBlockSize in the middle. What are the correct functions to call? Thanks, Geoffrey [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Cannot change block size 1 to 3! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving Wed Jun 13 17:51:34 2012 [0]PETSC ERROR: Libraries linked from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python --with-debugging=0 --with-c-support=1 --with-c++-support=1 --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 --PETSC_ARCH=darwin --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc --with-cc=/opt/local/bin/openmpicc --with-cxx=/opt/local/bin/openmpicxx --with-mpiexec=/opt/local/bin/openmpiexec --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 --download-ml --download-metis --download-parmetis --download-blacs --download-scalapack --download-mumps --download-ptscotch --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscLayoutSetBlockSize() line 460 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/vec/vec/impls/mpi/pmap.c [0]PETSC ERROR: MatSetBlockSize() line 6691 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: ! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving Wed Jun 13 17:51:34 2012 [0]PETSC ERROR: Libraries linked from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python --with-debugging=0 --with-c-support=1 --with-c++-support=1 --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 --PETSC_ARCH=darwin --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc --with-cc=/opt/local/bin/openmpicc --with-cxx=/opt/local/bin/openmpicxx --with-mpiexec=/opt/local/bin/openmpiexec --with-lapack-lib=/usr/lib/liblapack.dylib --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 --download-ml --download-metis --download-parmetis --download-blacs --download-scalapack --download-mumps --download-ptscotch --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscSolidMat() line 114 in petsc/ksp.cpp From jedbrown at mcs.anl.gov Wed Jun 13 20:03:30 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 13 Jun 2012 20:03:30 -0500 Subject: [petsc-users] MatSetBlockSize no longer works after MatCreateSeqAIJ In-Reply-To: References: Message-ID: What I do is MatCreate(comm,&A); MatSetSizes(A,...); MatSetBlockSize(A,bs); MatSetOptionsPrefix(A,"yourchoice_"); MatSetFromOptions(A); MatXAIJSetPreallocation(A,...); This should work with any matrix type, parallel and sequential, and you can specify it without interfering with any other matrices using -yourchoice_mat_type aij {baij, sbaij}. Devs, is there a reason that block size wasn't added to the convenience constructor? Wasn't there a discussion about adding it? On Wed, Jun 13, 2012 at 7:55 PM, Geoffrey Irving wrote: > Hello, > > The following code used to work, but breaks after upgrading to 3.3: > > .... > CHECK(MatCreateSeqAIJ(comm,3*n,3*n,0,scalar_view(nnz).data(),&mat)); > CHECK(MatSetBlockSize(mat,3)); > ... > > The error follows. Presumably MatSetBlockSize got more strict, and I > need to break the single MatCreateSeqAIJ into multiple lines (matrix > creation followed by preallocation) in order to insert MatSetBlockSize > in the middle. What are the correct functions to call? > > Thanks, > Geoffrey > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Cannot change block size 1 to 3! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 > 14:20:42 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving > Wed Jun 13 17:51:34 2012 > [0]PETSC ERROR: Libraries linked from > > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib > [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 > [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python > --with-debugging=0 --with-c-support=1 --with-c++-support=1 > --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 > --PETSC_ARCH=darwin > > --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc > --with-cc=/opt/local/bin/openmpicc > --with-cxx=/opt/local/bin/openmpicxx > --with-mpiexec=/opt/local/bin/openmpiexec > --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 > --download-ml --download-metis --download-parmetis --download-blacs > --download-scalapack --download-mumps --download-ptscotch > --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscLayoutSetBlockSize() line 460 in > > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/vec/vec/impls/mpi/pmap.c > [0]PETSC ERROR: MatSetBlockSize() line 6691 in > > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/mat/interface/matrix.c > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: ! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 > 14:20:42 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving > Wed Jun 13 17:51:34 2012 > [0]PETSC ERROR: Libraries linked from > > /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib > [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 > [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python > --with-debugging=0 --with-c-support=1 --with-c++-support=1 > --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 > --PETSC_ARCH=darwin > > --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc > --with-cc=/opt/local/bin/openmpicc > --with-cxx=/opt/local/bin/openmpicxx > --with-mpiexec=/opt/local/bin/openmpiexec > --with-lapack-lib=/usr/lib/liblapack.dylib > --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 > --download-ml --download-metis --download-parmetis --download-blacs > --download-scalapack --download-mumps --download-ptscotch > --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscSolidMat() line 114 in petsc/ksp.cpp > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bishtg at ornl.gov Wed Jun 13 18:41:46 2012 From: bishtg at ornl.gov (Bisht, Gautam) Date: Wed, 13 Jun 2012 19:41:46 -0400 Subject: [petsc-users] Question regadring VecScatter Message-ID: <3174ECA6-5656-419C-958C-004EE8DBF519@ornl.gov> Hi, I'm getting an error while creating a VecScatter for a MPI vector to another MPI vector. 'vec_from' has 3 elements (1 on proc-0 and 2 on proc-1) 'vec_to' has 3 elements (2 on proc-0 and 1 on proc-1) I want to scatter forward elements of 'vec_from' to 'vec_to' and am using IS to create the VecScatterCreate. Attached is the code. mpiexec -n 2 ./vecscatter_test produces [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Nonconforming object sizes! [0]PETSC ERROR: Local scatter sizes don't match! [0]PETSC ERROR: ------------------------------------------------------------------------ Any ideas what's going wrong? Thanks, -Gautam. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: vecscatter_test.F90 URL: From stali at geology.wisc.edu Wed Jun 13 22:02:53 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Wed, 13 Jun 2012 22:02:53 -0500 Subject: [petsc-users] Question regadring VecScatter In-Reply-To: <3174ECA6-5656-419C-958C-004EE8DBF519@ornl.gov> References: <3174ECA6-5656-419C-958C-004EE8DBF519@ornl.gov> Message-ID: <4FD9545D.6080400@geology.wisc.edu> I think ncells_from and ncells_to have to be the same size On 06/13/2012 06:41 PM, Bisht, Gautam wrote: > Hi, > > I'm getting an error while creating a VecScatter for a MPI vector to another MPI vector. > > 'vec_from' has 3 elements (1 on proc-0 and 2 on proc-1) > 'vec_to' has 3 elements (2 on proc-0 and 1 on proc-1) > > I want to scatter forward elements of 'vec_from' to 'vec_to' and am using IS to create the VecScatterCreate. Attached is the code. > > mpiexec -n 2 ./vecscatter_test > produces > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Nonconforming object sizes! > [0]PETSC ERROR: Local scatter sizes don't match! > [0]PETSC ERROR: ------------------------------------------------------------------------ > Any ideas what's going wrong? > > Thanks, > -Gautam. From thomas.witkowski at tu-dresden.de Thu Jun 14 03:19:06 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 14 Jun 2012 10:19:06 +0200 Subject: [petsc-users] Can IS be destroyed after set to PCFieldSplit Message-ID: <4FD99E7A.5010107@tu-dresden.de> Can the IS object be destroyed after it is used to initialize a FieldSplit? Or should it be destroyed after the corresponding FieldSplit has been destroyed? Thoams From knepley at gmail.com Thu Jun 14 06:14:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jun 2012 19:14:57 +0800 Subject: [petsc-users] Computation to communication ratio for PETSc's routines In-Reply-To: References: <4FD86F36.2030705@gfz-potsdam.de> <4FD89302.2070708@gfz-potsdam.de> Message-ID: On Thu, Jun 14, 2012 at 3:09 AM, Barry Smith wrote: > > On Jun 13, 2012, at 8:17 AM, Alexander Grayver wrote: > > > On 13.06.2012 13:24, Matthew Knepley wrote: > >> On Wed, Jun 13, 2012 at 6:45 PM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> Hello, > >> > >> As stated, I would like to estimate computation to communication ratio > for PETSc's linear algebra routines (e.g. MatMult{Transpose}, MatPtAP, > MatMatMult etc.) > >> As far as I understand the ratio depends on particular implementation > and number of processes one runs application on. > >> So I guess these formulas should be known? > >> > >> It depends on the input data. For example, block diagonal matrices do > not communicate in MatMult(). > >> > >> If not then I see one way to estimate it. Write app with those > operations, parse -log_summary and then divide flops/messages, however the > question is what information from -log_summary output should be used for > that? > >> > >> We give the number of messages and the average message length, so you > can get the total length across processes. Since > >> each side records, we divide by 2 (I think, you should check > PetscLogView for specifics). You can also use PetscLogViewPython() > >> to simplify parsing. > > > > Is there total amount of flops given for different operations separately > (I see max flops over all processes only)? > > Sadly no. My original plan was that people who wanted different summary > information would copy PetscLogView() and modify it to output just what > they want. Sadly that may be too difficult. You can back out the total flops from the flop rate and time. Matt > > Barry > > > > >> > >> Matt > >> > >> Thanks. > >> > >> -- > >> Regards, > >> Alexander > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> -- Norbert Wiener > > > > > > -- > > Regards, > > Alexander > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 14 06:19:46 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jun 2012 19:19:46 +0800 Subject: [petsc-users] Can IS be destroyed after set to PCFieldSplit In-Reply-To: <4FD99E7A.5010107@tu-dresden.de> References: <4FD99E7A.5010107@tu-dresden.de> Message-ID: On Thu, Jun 14, 2012 at 4:19 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Can the IS object be destroyed after it is used to initialize a > FieldSplit? Or should it be destroyed after the corresponding FieldSplit > has been destroyed? > Destroy it directly after. Matt > Thoams > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Thu Jun 14 06:24:28 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Thu, 14 Jun 2012 13:24:28 +0200 Subject: [petsc-users] Computation to communication ratio for PETSc's routines In-Reply-To: References: <4FD86F36.2030705@gfz-potsdam.de> <4FD89302.2070708@gfz-potsdam.de> Message-ID: <4FD9C9EC.7060801@gfz-potsdam.de> On 14.06.2012 13:14, Matthew Knepley wrote: > On Thu, Jun 14, 2012 at 3:09 AM, Barry Smith > wrote: > > > > Is there total amount of flops given for different operations > separately (I see max flops over all processes only)? > > Sadly no. My original plan was that people who wanted different > summary information would copy PetscLogView() and modify it to > output just what they want. Sadly that may be too difficult. > > > You can back out the total flops from the flop rate and time. I think I will write small apps with just one operation and then use the table that comes in the very beginning of the "PETSc Performance Summary" section. As far as I understand it contains data for the whole program and if there are no other operations I can get correct result? -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Thu Jun 14 08:01:33 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 14 Jun 2012 15:01:33 +0200 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) Message-ID: <4FD9E0AD.7040708@tu-dresden.de> I try to implement an efficient solver for my FEM based unsteady Navier Stokes code. The scenario I consider is realy simple: 2D FEM with Taylor-Hood element, equi-spaced grid, simple channel flow with prescribed inflow boundaries for the velocity, no boundary conditions for the pressure. Because I want to solve unsteady flows, at each timepoint an Ossen problem is solved (implicit Euler time discretization, "trivial" linearization). For using PETSc, I created a fieldsplit preconditioner, that is configured in the following way: mpirun -np 2 ./navier_stokes init/channel_parallel.dat.2d \ -ksp_type fgmres \ -pc_type fieldsplit \ -pc_fieldsplit_type SCHUR \ -pc_fieldsplit_schur_fact_type FULL \ -ksp_converged_reason \ -ksp_monitor_true_residual \ -fieldsplit_velocity_ksp_type preonly \ -fieldsplit_velocity_pc_type lu \ -fieldsplit_velocity_pc_factor_mat_solver_package mumps \ -fieldsplit_pressure_ksp_type gmres \ -fieldsplit_pressure_ksp_max_it 10 \ -fieldsplit_pressure_ksp_rtol 1.0e-2 \ -fieldsplit_pressure_pc_type lsc \ -fieldsplit_pressure_lsc_ksp_type bcgs \ -fieldsplit_pressure_lsc_ksp_max_it 10 \ -fieldsplit_pressure_lsc_ksp_rtol 1.0e-2 \ -fieldsplit_pressure_lsc_pc_type hypre The direct solver for the velocity part is just for debugging and should be replaced when everything else works fine. The point is, that I found this constellation to be not very efficient. It takes around 20 to 30 iterations, which takes around 30 seconds on a very small problem size (around 20000 global unknows for each velocity component and 5000 global unknowns for the pressure) on a very fast desktop CPU (some new Intel Xeno with 4 core). Any hints for improvements? Thomas From knepley at gmail.com Thu Jun 14 08:06:48 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jun 2012 21:06:48 +0800 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) In-Reply-To: <4FD9E0AD.7040708@tu-dresden.de> References: <4FD9E0AD.7040708@tu-dresden.de> Message-ID: On Thu, Jun 14, 2012 at 9:01 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > I try to implement an efficient solver for my FEM based unsteady Navier > Stokes code. The scenario I consider is realy simple: 2D FEM with > Taylor-Hood element, equi-spaced grid, simple channel flow with prescribed > inflow boundaries for the velocity, no boundary conditions for the > pressure. Because I want to With no pressure BC, you need to specify the null space. Look at SNES ex62. > solve unsteady flows, at each timepoint an Ossen problem is solved > (implicit Euler time discretization, "trivial" linearization). For using > PETSc, I created a fieldsplit preconditioner, that is configured in the > following way: > > mpirun -np 2 ./navier_stokes init/channel_parallel.dat.2d \ > -ksp_type fgmres \ > -pc_type fieldsplit \ > -pc_fieldsplit_type SCHUR \ > -pc_fieldsplit_schur_fact_type FULL \ > -ksp_converged_reason \ > -ksp_monitor_true_residual \ > -fieldsplit_velocity_ksp_type preonly \ > -fieldsplit_velocity_pc_type lu \ > -fieldsplit_velocity_pc_**factor_mat_solver_package mumps \ > -fieldsplit_pressure_ksp_type gmres \ > -fieldsplit_pressure_ksp_max_**it 10 \ > -fieldsplit_pressure_ksp_rtol 1.0e-2 \ > -fieldsplit_pressure_pc_type lsc \ > This makes no sense unless you specify an auxiliary operator. Just leave it at jacobi. When you use LU for velocity, it will converge in 1 iterate. Since it doesn't, it means you have a problem with the null space. Matt > -fieldsplit_pressure_lsc_ksp_**type bcgs \ > -fieldsplit_pressure_lsc_ksp_**max_it 10 \ > -fieldsplit_pressure_lsc_ksp_**rtol 1.0e-2 \ > -fieldsplit_pressure_lsc_pc_**type hypre > > The direct solver for the velocity part is just for debugging and should > be replaced when everything else works fine. The point is, that I found > this constellation to be not very efficient. It takes around 20 to 30 > iterations, which takes around 30 seconds on a very small problem size > (around 20000 global unknows for each velocity component and 5000 global > unknowns for the pressure) on a very fast desktop CPU (some new Intel Xeno > with 4 core). Any hints for improvements? > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bishtg at ornl.gov Thu Jun 14 08:07:50 2012 From: bishtg at ornl.gov (Bisht, Gautam) Date: Thu, 14 Jun 2012 09:07:50 -0400 Subject: [petsc-users] Question regadring VecScatter In-Reply-To: <4FD9545D.6080400@geology.wisc.edu> References: <3174ECA6-5656-419C-958C-004EE8DBF519@ornl.gov> <4FD9545D.6080400@geology.wisc.edu> Message-ID: Thanks, Tabrez. The local size of IS (is_from and is_to) need to of same length. -Gautam. On Jun 13, 2012, at 11:02 PM, Tabrez Ali wrote: > I think ncells_from and ncells_to have to be the same size > > On 06/13/2012 06:41 PM, Bisht, Gautam wrote: >> Hi, >> >> I'm getting an error while creating a VecScatter for a MPI vector to another MPI vector. >> >> 'vec_from' has 3 elements (1 on proc-0 and 2 on proc-1) >> 'vec_to' has 3 elements (2 on proc-0 and 1 on proc-1) >> >> I want to scatter forward elements of 'vec_from' to 'vec_to' and am using IS to create the VecScatterCreate. Attached is the code. >> >> mpiexec -n 2 ./vecscatter_test >> produces >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Nonconforming object sizes! >> [0]PETSC ERROR: Local scatter sizes don't match! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> Any ideas what's going wrong? >> >> Thanks, >> -Gautam. > From stali at geology.wisc.edu Thu Jun 14 08:56:05 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 14 Jun 2012 08:56:05 -0500 Subject: [petsc-users] Question regadring VecScatter In-Reply-To: References: <3174ECA6-5656-419C-958C-004EE8DBF519@ornl.gov> <4FD9545D.6080400@geology.wisc.edu> Message-ID: <4FD9ED75.5070600@geology.wisc.edu> Yes. See Figure 11 in the manual and the accompanying text at the bottom. On 06/14/2012 08:07 AM, Bisht, Gautam wrote: > Thanks, Tabrez. > > The local size of IS (is_from and is_to) need to of same length. > > -Gautam. > > On Jun 13, 2012, at 11:02 PM, Tabrez Ali wrote: > >> I think ncells_from and ncells_to have to be the same size >> >> On 06/13/2012 06:41 PM, Bisht, Gautam wrote: >>> Hi, >>> >>> I'm getting an error while creating a VecScatter for a MPI vector to another MPI vector. >>> >>> 'vec_from' has 3 elements (1 on proc-0 and 2 on proc-1) >>> 'vec_to' has 3 elements (2 on proc-0 and 1 on proc-1) >>> >>> I want to scatter forward elements of 'vec_from' to 'vec_to' and am using IS to create the VecScatterCreate. Attached is the code. >>> >>> mpiexec -n 2 ./vecscatter_test >>> produces >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Nonconforming object sizes! >>> [0]PETSC ERROR: Local scatter sizes don't match! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> Any ideas what's going wrong? >>> >>> Thanks, >>> -Gautam. -- No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley From thomas.witkowski at tu-dresden.de Thu Jun 14 09:07:23 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 14 Jun 2012 16:07:23 +0200 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) In-Reply-To: References: <4FD9E0AD.7040708@tu-dresden.de> Message-ID: <4FD9F01B.805@tu-dresden.de> Am 14.06.2012 15:06, schrieb Matthew Knepley: > On Thu, Jun 14, 2012 at 9:01 PM, Thomas Witkowski > > wrote: > > I try to implement an efficient solver for my FEM based unsteady > Navier Stokes code. The scenario I consider is realy simple: 2D > FEM with Taylor-Hood element, equi-spaced grid, simple channel > flow with prescribed inflow boundaries for the velocity, no > boundary conditions for the pressure. Because I want to > > > With no pressure BC, you need to specify the null space. Look at SNES > ex62. As so often, I forgotten about this :) As I have free outflow boundary for the velocity (so no Dirichlet boundary), I think, the nullspace must be somehow different than just setting a constant on all pressure nodes. So I want back to the even more simple case, the driven cavity scenario. Here I have dirichlet BC on all velocity components and boundaries and no pressure BC. There is no influence on the solver, whether I set the null space or not (I checked for it with -ksp_view that there is a null space set). Only when going from fgmres to gmres for the outer solver, setting the null space has some (positive) change in the solver convergence. > solve unsteady flows, at each timepoint an Ossen problem is solved > (implicit Euler time discretization, "trivial" linearization). For > using PETSc, I created a fieldsplit preconditioner, that is > configured in the following way: > > mpirun -np 2 ./navier_stokes init/channel_parallel.dat.2d \ > -ksp_type fgmres \ > -pc_type fieldsplit \ > -pc_fieldsplit_type SCHUR \ > -pc_fieldsplit_schur_fact_type FULL \ > -ksp_converged_reason \ > -ksp_monitor_true_residual \ > -fieldsplit_velocity_ksp_type preonly \ > -fieldsplit_velocity_pc_type lu \ > -fieldsplit_velocity_pc_factor_mat_solver_package mumps \ > -fieldsplit_pressure_ksp_type gmres \ > -fieldsplit_pressure_ksp_max_it 10 \ > -fieldsplit_pressure_ksp_rtol 1.0e-2 \ > -fieldsplit_pressure_pc_type lsc \ > > > This makes no sense unless you specify an auxiliary operator. Just > leave it at jacobi. When you use LU > for velocity, it will converge in 1 iterate. Since it doesn't, it > means you have a problem with the null space. So even for the driven cavity, the null space is set, and again I use the same solver, it needs something like 10 or 15 iterations. I don't understand your argument, that using LU for the velocity part, this solver should converge in 1 iterations. It still has an inexact solver for the Schur complement and the solver for the composite matrix in LSC is also inexact. Thomas > > Matt > > -fieldsplit_pressure_lsc_ksp_type bcgs \ > -fieldsplit_pressure_lsc_ksp_max_it 10 \ > -fieldsplit_pressure_lsc_ksp_rtol 1.0e-2 \ > -fieldsplit_pressure_lsc_pc_type hypre > > The direct solver for the velocity part is just for debugging and > should be replaced when everything else works fine. The point is, > that I found this constellation to be not very efficient. It takes > around 20 to 30 iterations, which takes around 30 seconds on a > very small problem size (around 20000 global unknows for each > velocity component and 5000 global unknowns for the pressure) on a > very fast desktop CPU (some new Intel Xeno with 4 core). Any hints > for improvements? > > Thomas > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 14 09:20:50 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 14 Jun 2012 22:20:50 +0800 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) In-Reply-To: <4FD9F01B.805@tu-dresden.de> References: <4FD9E0AD.7040708@tu-dresden.de> <4FD9F01B.805@tu-dresden.de> Message-ID: On Thu, Jun 14, 2012 at 10:07 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Am 14.06.2012 15:06, schrieb Matthew Knepley: > > On Thu, Jun 14, 2012 at 9:01 PM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> I try to implement an efficient solver for my FEM based unsteady Navier >> Stokes code. The scenario I consider is realy simple: 2D FEM with >> Taylor-Hood element, equi-spaced grid, simple channel flow with prescribed >> inflow boundaries for the velocity, no boundary conditions for the >> pressure. Because I want to > > > With no pressure BC, you need to specify the null space. Look at SNES > ex62. > > As so often, I forgotten about this :) As I have free outflow boundary for > the velocity (so no Dirichlet boundary), I think, the nullspace must be > somehow different than just setting a constant on all pressure nodes. So I > want back to the even more simple case, the driven cavity scenario. Here I > have dirichlet BC on all velocity components and boundaries and no pressure > BC. There is no influence on the solver, whether I set the null space or > not (I checked for it with -ksp_view that there is a null space set). Only > when going from fgmres to gmres for the outer solver, setting the null > space has some (positive) change in the solver convergence. > It has a null space, so whatever you observe in a single run is an outlier. Try removing the null space in ex62. It will fail to converge. > > > solve unsteady flows, at each timepoint an Ossen problem is solved >> (implicit Euler time discretization, "trivial" linearization). For using >> PETSc, I created a fieldsplit preconditioner, that is configured in the >> following way: >> >> mpirun -np 2 ./navier_stokes init/channel_parallel.dat.2d \ >> -ksp_type fgmres \ >> -pc_type fieldsplit \ >> -pc_fieldsplit_type SCHUR \ >> -pc_fieldsplit_schur_fact_type FULL \ >> -ksp_converged_reason \ >> -ksp_monitor_true_residual \ >> -fieldsplit_velocity_ksp_type preonly \ >> -fieldsplit_velocity_pc_type lu \ >> -fieldsplit_velocity_pc_factor_mat_solver_package mumps \ >> -fieldsplit_pressure_ksp_type gmres \ >> -fieldsplit_pressure_ksp_max_it 10 \ >> -fieldsplit_pressure_ksp_rtol 1.0e-2 \ >> -fieldsplit_pressure_pc_type lsc \ >> > > This makes no sense unless you specify an auxiliary operator. Just leave > it at jacobi. When you use LU > for velocity, it will converge in 1 iterate. Since it doesn't, it means > you have a problem with the null space. > > So even for the driven cavity, the null space is set, and again I use the > same solver, it needs something like 10 or 15 iterations. I don't > understand your argument, that using LU for the velocity part, this solver > should converge in 1 iterations. It still has an inexact solver for the > Schur complement and the solver for the composite matrix in LSC is also > inexact. > 1) Start with -fieldsplit_pressure_ksp_rtol 1e-10. It will converge in 1 iterate. Back off the Schur tolerance to balance that cost with the cost of extra iterates. This is the game. 2) Again, LSC is not doing anything for you right now, unless you are indeed giving an auxiliary operator. Use jacobi. Matt > Thomas > > > Matt > > >> -fieldsplit_pressure_lsc_ksp_type bcgs \ >> -fieldsplit_pressure_lsc_ksp_max_it 10 \ >> -fieldsplit_pressure_lsc_ksp_rtol 1.0e-2 \ >> -fieldsplit_pressure_lsc_pc_type hypre >> >> The direct solver for the velocity part is just for debugging and should >> be replaced when everything else works fine. The point is, that I found >> this constellation to be not very efficient. It takes around 20 to 30 >> iterations, which takes around 30 seconds on a very small problem size >> (around 20000 global unknows for each velocity component and 5000 global >> unknowns for the pressure) on a very fast desktop CPU (some new Intel Xeno >> with 4 core). Any hints for improvements? >> >> Thomas >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 14 10:43:47 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 14 Jun 2012 10:43:47 -0500 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) In-Reply-To: References: <4FD9E0AD.7040708@tu-dresden.de> <4FD9F01B.805@tu-dresden.de> Message-ID: On Thu, Jun 14, 2012 at 9:20 AM, Matthew Knepley wrote: > 2) Again, LSC is not doing anything for you right now, unless you are > indeed giving an auxiliary operator. Use jacobi. What do you mean by this? LSC just uses the block structure; it's not like PCD or other user-defined Schur preconditioners. Note that it generally requires -ksp_diagonal_scale (see src/ksp/ksp/examples/tests/ex11.c, especially the options used for runex11 and runex11_2). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Thu Jun 14 13:09:20 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Thu, 14 Jun 2012 14:09:20 -0400 (EDT) Subject: [petsc-users] ml precondioner Message-ID: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> Hello I have a 2 dimensional Poisson problem and the Laplacian is defined on a mesh that has both uniform and non-uniform regimes. I am using the following command: -pc_type ml -mg_levels_ksp_type gmres -mg_levels_pc_type lu This gives a good result. (However, if I don't specify the -mg_level_ksp_type and -mg_levels_pc_type to gmres and lu respectively, the default are used (richardson and sor respectively). These lead to wrong results and the solver is extremely slow.) Now, the following method: -ksp_type gmres -pc_type lu gives a slightly better result and is faster. The speed difference is quite conspicuous for a 501 x 1001 grid size. According to my limited knowledge, asymmetric matrices are better dealt with by gmres and lu. However, multigrid methods are supposed to have O(n) complexity. Why is the non-multigrid method doing better in both speed and accuracy? Any input would be greatly appreciated. Thank you. -- Nakib :) From irving at naml.us Thu Jun 14 13:24:17 2012 From: irving at naml.us (Geoffrey Irving) Date: Thu, 14 Jun 2012 11:24:17 -0700 Subject: [petsc-users] MatSetBlockSize no longer works after MatCreateSeqAIJ In-Reply-To: References: Message-ID: Thanks, that works. MatSetOptionsPrefix is convenient. Geoffrey On Wed, Jun 13, 2012 at 6:03 PM, Jed Brown wrote: > What I do is > > MatCreate(comm,&A); > MatSetSizes(A,...); > MatSetBlockSize(A,bs); > MatSetOptionsPrefix(A,"yourchoice_"); > MatSetFromOptions(A); > MatXAIJSetPreallocation(A,...); > > This should work with any matrix type, parallel and sequential, and you can > specify it without interfering with any other matrices using > -yourchoice_mat_type aij {baij, sbaij}. > > > Devs, is there a reason that block size wasn't added to the convenience > constructor? Wasn't there a discussion about adding it? > > On Wed, Jun 13, 2012 at 7:55 PM, Geoffrey Irving wrote: >> >> Hello, >> >> The following code used to work, but breaks after upgrading to 3.3: >> >> ? ?.... >> ? ?CHECK(MatCreateSeqAIJ(comm,3*n,3*n,0,scalar_view(nnz).data(),&mat)); >> ? ?CHECK(MatSetBlockSize(mat,3)); >> ? ?... >> >> The error follows. ?Presumably MatSetBlockSize got more strict, and I >> need to break the single MatCreateSeqAIJ into multiple lines (matrix >> creation followed by preallocation) in order to insert MatSetBlockSize >> in the middle. ?What are the correct functions to call? >> >> Thanks, >> Geoffrey >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Arguments are incompatible! >> [0]PETSC ERROR: Cannot change block size 1 to 3! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun ?5 >> 14:20:42 CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving >> Wed Jun 13 17:51:34 2012 >> [0]PETSC ERROR: Libraries linked from >> >> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib >> [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 >> [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python >> --with-debugging=0 --with-c-support=1 --with-c++-support=1 >> --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 >> --PETSC_ARCH=darwin >> >> --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc >> --with-cc=/opt/local/bin/openmpicc >> --with-cxx=/opt/local/bin/openmpicxx >> --with-mpiexec=/opt/local/bin/openmpiexec >> --with-lapack-lib=/usr/lib/liblapack.dylib >> --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 >> --download-ml --download-metis --download-parmetis --download-blacs >> --download-scalapack --download-mumps --download-ptscotch >> --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscLayoutSetBlockSize() line 460 in >> >> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/vec/vec/impls/mpi/pmap.c >> [0]PETSC ERROR: MatSetBlockSize() line 6691 in >> >> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/petsc-3.3-p0/src/mat/interface/matrix.c >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Arguments are incompatible! >> [0]PETSC ERROR: ?! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun ?5 >> 14:20:42 CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: Unknown Name on a darwin named tile.local by irving >> Wed Jun 13 17:51:34 2012 >> [0]PETSC ERROR: Libraries linked from >> >> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc/lib >> [0]PETSC ERROR: Configure run at Wed Jun 13 12:11:05 2012 >> [0]PETSC ERROR: Configure options --prefix=/opt/local --with-python >> --with-debugging=0 --with-c-support=1 --with-c++-support=1 >> --with-pic=fPIC --with-shared-libraries=0 --with-mpi=1 >> --PETSC_ARCH=darwin >> >> --prefix=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_petsc/petsc/work/destroot/opt/local/lib/petsc >> --with-cc=/opt/local/bin/openmpicc >> --with-cxx=/opt/local/bin/openmpicxx >> --with-mpiexec=/opt/local/bin/openmpiexec >> --with-lapack-lib=/usr/lib/liblapack.dylib >> --with-blas-lib=/usr/lib/libblas.dylib --download-umfpack=1 >> --download-ml --download-metis --download-parmetis --download-blacs >> --download-scalapack --download-mumps --download-ptscotch >> --with-fc=/opt/local/bin/openmpif90 --LIBS=-lstdc++ >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> [0]PETSC ERROR: PetscSolidMat() line 114 in petsc/ksp.cpp > > From li at loyno.edu Thu Jun 14 14:08:29 2012 From: li at loyno.edu (Xuefeng Li) Date: Thu, 14 Jun 2012 14:08:29 -0500 (CDT) Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: Hi, all. I am using a multilevel DMMG in my petsc 3.1-p8 program. The program calls SNES_KSPSolve(), and in turn calls KSP_Solve(). The number of times KSP_Solve() is called seems to be indefinite (10's of thousands of times). Are there any database keys to control the number of times KSP_Solve() is called? Where is the loop that calls KSP_Solve? The key -ksp_max_it does not seem to affect it. Regards, --Xuefeng Li, (504)865-3340(phone) Like floating clouds, the heart rests easy Like flowing water, the spirit stays free http://www.loyno.edu/~li/home New Orleans, Louisiana (504)865-2051(fax) From knepley at gmail.com Thu Jun 14 14:15:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 03:15:44 +0800 Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: On Fri, Jun 15, 2012 at 3:08 AM, Xuefeng Li
  • wrote: > Hi, all. > > I am using a multilevel DMMG in my petsc 3.1-p8 > program. The program calls SNES_KSPSolve(), and > in turn calls KSP_Solve(). The number of times > KSP_Solve() is called seems to be indefinite > (10's of thousands of times). > The default number of iterates is 10K. > Are there any database keys to control the number of > times KSP_Solve() is called? Where is the loop that > calls KSP_Solve? The key -ksp_max_it does not seem > to affect it. > Then this option did not reach the solver. You may have set a prefix. Run with -snes_max_it 1 -ksp_max_it 1 -snes_view to see what you are actually doing. Matt > Regards, > > --Xuefeng Li, (504)865-3340(phone) > Like floating clouds, the heart rests easy > Like flowing water, the spirit stays free > http://www.loyno.edu/~li/home > New Orleans, Louisiana (504)865-2051(fax) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 14 14:16:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 14 Jun 2012 14:16:29 -0500 Subject: [petsc-users] ml precondioner In-Reply-To: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> References: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> Message-ID: On Thu, Jun 14, 2012 at 1:09 PM, Nakib Haider Protik wrote: > I have a 2 dimensional Poisson problem and the Laplacian is defined on a > mesh that has both uniform and non-uniform regimes. I am using the > following command: > > -pc_type ml -mg_levels_ksp_type gmres -mg_levels_pc_type lu > This does a direct solve on every level, of course it's going to be slow. > > This gives a good result. (However, if I don't specify the > -mg_level_ksp_type and -mg_levels_pc_type to gmres and lu respectively, > the default are used (richardson and sor respectively). These lead to > wrong results and the solver is extremely slow.) > Are you using petsc-3.3? Run with -mg_levels_ksp_type chebyshev (and only that). Does that converge better? > > Now, the following method: > > -ksp_type gmres -pc_type lu > > gives a slightly better result and is faster. The speed difference is > quite conspicuous for a 501 x 1001 grid size. > > According to my limited knowledge, asymmetric matrices are better dealt > with by gmres and lu. However, multigrid methods are supposed to have O(n) > complexity. Why is the non-multigrid method doing better in both speed and > accuracy? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nprot048 at uottawa.ca Thu Jun 14 14:35:26 2012 From: nprot048 at uottawa.ca (Nakib Haider Protik) Date: Thu, 14 Jun 2012 15:35:26 -0400 (EDT) Subject: [petsc-users] ml precondioner In-Reply-To: References: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> Message-ID: <36361.137.122.149.22.1339702526.squirrel@webmail01.uottawa.ca> I am running 3.0.0-p12. Upgrading to 3.3 has proven difficult for me. In my version I used -mg_levels_ksp_type chebychev and the error is greater than both ordinary gmres with lu, and ml with gmres and lu on each level. Also, it takes a very long time to converge for even a 200 x 200 grid. Thanks. > On Thu, Jun 14, 2012 at 1:09 PM, Nakib Haider Protik > wrote: > >> I have a 2 dimensional Poisson problem and the Laplacian is defined on a >> mesh that has both uniform and non-uniform regimes. I am using the >> following command: >> >> -pc_type ml -mg_levels_ksp_type gmres -mg_levels_pc_type lu >> > > This does a direct solve on every level, of course it's going to be slow. > > >> >> This gives a good result. (However, if I don't specify the >> -mg_level_ksp_type and -mg_levels_pc_type to gmres and lu respectively, >> the default are used (richardson and sor respectively). These lead to >> wrong results and the solver is extremely slow.) >> > > Are you using petsc-3.3? > > Run with -mg_levels_ksp_type chebyshev (and only that). Does that converge > better? > > >> >> Now, the following method: >> >> -ksp_type gmres -pc_type lu >> >> gives a slightly better result and is faster. The speed difference is >> quite conspicuous for a 501 x 1001 grid size. >> >> According to my limited knowledge, asymmetric matrices are better dealt >> with by gmres and lu. However, multigrid methods are supposed to have >> O(n) >> complexity. Why is the non-multigrid method doing better in both speed >> and >> accuracy? >> > -- Nakib :) From knepley at gmail.com Thu Jun 14 14:38:11 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 03:38:11 +0800 Subject: [petsc-users] ml precondioner In-Reply-To: <36361.137.122.149.22.1339702526.squirrel@webmail01.uottawa.ca> References: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> <36361.137.122.149.22.1339702526.squirrel@webmail01.uottawa.ca> Message-ID: On Fri, Jun 15, 2012 at 3:35 AM, Nakib Haider Protik wrote: > I am running 3.0.0-p12. Upgrading to 3.3 has proven difficult for me. > This is 5 years old. You will notice that we have an email list to help people with upgrade problems. We have not received any mails about problems. > In my version I used -mg_levels_ksp_type chebychev and the error is > greater than both ordinary gmres with lu, and ml with gmres and lu on each > The solver has nothing to do with the error. This is a matter for the iterative tolerance and condition number of the problem. Matt > level. Also, it takes a very long time to converge for even a 200 x 200 > grid. > > Thanks. > > > On Thu, Jun 14, 2012 at 1:09 PM, Nakib Haider Protik > > wrote: > > > >> I have a 2 dimensional Poisson problem and the Laplacian is defined on a > >> mesh that has both uniform and non-uniform regimes. I am using the > >> following command: > >> > >> -pc_type ml -mg_levels_ksp_type gmres -mg_levels_pc_type lu > >> > > > > This does a direct solve on every level, of course it's going to be slow. > > > > > >> > >> This gives a good result. (However, if I don't specify the > >> -mg_level_ksp_type and -mg_levels_pc_type to gmres and lu respectively, > >> the default are used (richardson and sor respectively). These lead to > >> wrong results and the solver is extremely slow.) > >> > > > > Are you using petsc-3.3? > > > > Run with -mg_levels_ksp_type chebyshev (and only that). Does that > converge > > better? > > > > > >> > >> Now, the following method: > >> > >> -ksp_type gmres -pc_type lu > >> > >> gives a slightly better result and is faster. The speed difference is > >> quite conspicuous for a 501 x 1001 grid size. > >> > >> According to my limited knowledge, asymmetric matrices are better dealt > >> with by gmres and lu. However, multigrid methods are supposed to have > >> O(n) > >> complexity. Why is the non-multigrid method doing better in both speed > >> and > >> accuracy? > >> > > > > > -- > Nakib :) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 14 14:38:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 14 Jun 2012 14:38:13 -0500 Subject: [petsc-users] ml precondioner In-Reply-To: <36361.137.122.149.22.1339702526.squirrel@webmail01.uottawa.ca> References: <36040.137.122.149.22.1339697360.squirrel@webmail01.uottawa.ca> <36361.137.122.149.22.1339702526.squirrel@webmail01.uottawa.ca> Message-ID: On Thu, Jun 14, 2012 at 2:35 PM, Nakib Haider Protik wrote: > I am running 3.0.0-p12. Upgrading to 3.3 has proven difficult for me. > Do it now. It should be easier, not harder. If you get stuck, send all the log files to petsc-maint. > > In my version I used -mg_levels_ksp_type chebychev and the error is > greater than both ordinary gmres with lu, and ml with gmres and lu on each > level. Also, it takes a very long time to converge for even a 200 x 200 > grid. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From li at loyno.edu Thu Jun 14 15:22:07 2012 From: li at loyno.edu (Xuefeng Li) Date: Thu, 14 Jun 2012 15:22:07 -0500 (CDT) Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: On Fri, 15 Jun 2012, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 3:08 AM, Xuefeng Li
  • wrote: > >> Are there any database keys to control the number of >> times KSP_Solve() is called? Where is the loop that >> calls KSP_Solve? The key -ksp_max_it does not seem >> to affect it. >> > > Then this option did not reach the solver. You may have set a prefix. > > Run with -snes_max_it 1 -ksp_max_it 1 -snes_view to see what you are > actually doing. > The key -ksp_max_it affects the iteration number for the (main) ksp object for SNES, but not the ksp object for PC. I observed yet another (related) problem. Even though I set KSP type to be GMRES, the type is GMRES only on the coarse level of the DMMG; the type changes to MG on all finer levels. How do I control KSP and PC on finer levels of DMMG? Regards, --Xuefeng Li, (504)865-3340(phone) Like floating clouds, the heart rests easy Like flowing water, the spirit stays free http://www.loyno.edu/~li/home New Orleans, Louisiana (504)865-2051(fax) From knepley at gmail.com Thu Jun 14 15:25:04 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 04:25:04 +0800 Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: On Fri, Jun 15, 2012 at 4:22 AM, Xuefeng Li
  • wrote: > On Fri, 15 Jun 2012, Matthew Knepley wrote: > > On Fri, Jun 15, 2012 at 3:08 AM, Xuefeng Li
  • wrote: >> >> Are there any database keys to control the number of >>> times KSP_Solve() is called? Where is the loop that >>> calls KSP_Solve? The key -ksp_max_it does not seem >>> to affect it. >>> >>> >> Then this option did not reach the solver. You may have set a prefix. >> >> Run with -snes_max_it 1 -ksp_max_it 1 -snes_view to see what you are >> actually doing. >> >> The key -ksp_max_it affects the iteration number > for the (main) ksp object for SNES, but not the ksp > object for PC. > 1) Why do you have a KSP object underneath your PC? 2) Since you do -sub_ksp_max_it 1 would be the default > I observed yet another (related) problem. Even though > I set KSP type to be GMRES, the type is GMRES only on > the coarse level of the DMMG; the type changes to MG > on all finer levels. How do I control KSP and PC on > finer levels of DMMG? > 3) Don't use DMMG, it has been replaced with something much simpler in the latest release There you can give -mg_levels_ksp_type Matt > > Regards, > > --Xuefeng Li, (504)865-3340(phone) > Like floating clouds, the heart rests easy > Like flowing water, the spirit stays free > http://www.loyno.edu/~li/home > New Orleans, Louisiana (504)865-2051(fax) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From li at loyno.edu Thu Jun 14 15:59:54 2012 From: li at loyno.edu (Xuefeng Li) Date: Thu, 14 Jun 2012 15:59:54 -0500 (CDT) Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: On Fri, 15 Jun 2012, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 4:22 AM, Xuefeng Li
  • wrote: > >> On Fri, 15 Jun 2012, Matthew Knepley wrote: >> >> On Fri, Jun 15, 2012 at 3:08 AM, Xuefeng Li
  • wrote: >>> >>> Are there any database keys to control the number of >>>> times KSP_Solve() is called? Where is the loop that >>>> calls KSP_Solve? The key -ksp_max_it does not seem >>>> to affect it. >>>> >>>> >>> Then this option did not reach the solver. You may have set a prefix. >>> >>> Run with -snes_max_it 1 -ksp_max_it 1 -snes_view to see what you are >>> actually doing. >>> >>> The key -ksp_max_it affects the iteration number >> for the (main) ksp object for SNES, but not the ksp >> object for PC. >> > > 1) Why do you have a KSP object underneath your PC? > I should've made myself clearer in the last email. I get ksp through DMMGGetKSP(), and get pc through KSPGetPC(). I then KSPSetType() to gmres; PCSetType() to pcasm. That's all I've done in the program. After using -snes_view, the output shows the following. KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=2222, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object:(sub_) type: preonly maximum iterations=10000, initial guess is zero Iteration remains 10,000. Regards, --Xuefeng Li, (504)865-3340(phone) Like floating clouds, the heart rests easy Like flowing water, the spirit stays free http://www.loyno.edu/~li/home New Orleans, Louisiana (504)865-2051(fax) From knepley at gmail.com Thu Jun 14 16:04:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 05:04:49 +0800 Subject: [petsc-users] Database key for KSP_Solve In-Reply-To: References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu> Message-ID: On Fri, Jun 15, 2012 at 4:59 AM, Xuefeng Li
  • wrote: > On Fri, 15 Jun 2012, Matthew Knepley wrote: > > On Fri, Jun 15, 2012 at 4:22 AM, Xuefeng Li
  • wrote: >> >> On Fri, 15 Jun 2012, Matthew Knepley wrote: >>> >>> On Fri, Jun 15, 2012 at 3:08 AM, Xuefeng Li
  • wrote: >>> >>>> >>>> Are there any database keys to control the number of >>>> >>>>> times KSP_Solve() is called? Where is the loop that >>>>> calls KSP_Solve? The key -ksp_max_it does not seem >>>>> to affect it. >>>>> >>>>> >>>>> Then this option did not reach the solver. You may have set a prefix. >>>> >>>> Run with -snes_max_it 1 -ksp_max_it 1 -snes_view to see what you are >>>> actually doing. >>>> >>>> The key -ksp_max_it affects the iteration number >>>> >>> for the (main) ksp object for SNES, but not the ksp >>> object for PC. >>> >>> >> 1) Why do you have a KSP object underneath your PC? >> >> I should've made myself clearer in the last email. I get > ksp through DMMGGetKSP(), and get pc through KSPGetPC(). > I then KSPSetType() to gmres; PCSetType() to pcasm. That's > all I've done in the program. > > After using -snes_view, the output shows the following. > KSP Object: > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=2222, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: > type: asm > Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 > Additive Schwarz: restriction/interpolation type - RESTRICT > Local solve is same for all blocks, in the following KSP and PC > objects: > KSP Object:(sub_) > ^^^^^^^^^^^^^^^^^^ It is telling you right here what the name is, -sub_ksp_max_it 1 Matt > type: preonly > maximum iterations=10000, initial guess is zero > > Iteration remains 10,000. > > > Regards, > > --Xuefeng Li, (504)865-3340(phone) > Like floating clouds, the heart rests easy > Like flowing water, the spirit stays free > http://www.loyno.edu/~li/home > New Orleans, Louisiana (504)865-2051(fax) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 14 17:07:21 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 14 Jun 2012 17:07:21 -0500 Subject: [petsc-users] How to improve this solver for Navier-Stokes equation (based on fieldsplit and lsc) In-Reply-To: <4FD9F01B.805@tu-dresden.de> References: <4FD9E0AD.7040708@tu-dresden.de> <4FD9F01B.805@tu-dresden.de> Message-ID: <13A2D552-B24B-4BAC-A30B-2032FC47D8FF@mcs.anl.gov> On Jun 14, 2012, at 9:07 AM, Thomas Witkowski wrote: > Am 14.06.2012 15:06, schrieb Matthew Knepley: >> On Thu, Jun 14, 2012 at 9:01 PM, Thomas Witkowski wrote: >> I try to implement an efficient solver for my FEM based unsteady Navier Stokes code. The scenario I consider is realy simple: 2D FEM with Taylor-Hood element, equi-spaced grid, simple channel flow with prescribed inflow boundaries for the velocity, no boundary conditions for the pressure. Because I want to >> >> With no pressure BC, you need to specify the null space. Look at SNES ex62. > As so often, I forgotten about this :) As I have free outflow boundary for the velocity (so no Dirichlet boundary), I think, the nullspace must be somehow different than just setting a constant on all pressure nodes. So I want back to the even more simple case, the driven cavity scenario. Here I have dirichlet BC on all velocity components and boundaries and no pressure BC. There is no influence on the solver, whether I set the null space or not (I checked for it with -ksp_view that there is a null space set). Only when going from fgmres to gmres for the outer solver, setting the null space has some (positive) change in the solver convergence. Since you have a gmres in your pressure solver you must use fgmres in the outer solver. You cannot use gmres inside gmres. Also you can use -fieldsplit_pressure_ksp_monitor_true_residual -fieldsplit_pressure_ksp_converged_reason etc to track the inner solvers also to get a more complete picture of what is happening in terms of convergence. Might be useful. Barry > >> >> solve unsteady flows, at each timepoint an Ossen problem is solved (implicit Euler time discretization, "trivial" linearization). For using PETSc, I created a fieldsplit preconditioner, that is configured in the following way: >> >> mpirun -np 2 ./navier_stokes init/channel_parallel.dat.2d \ >> -ksp_type fgmres \ >> -pc_type fieldsplit \ >> -pc_fieldsplit_type SCHUR \ >> -pc_fieldsplit_schur_fact_type FULL \ >> -ksp_converged_reason \ >> -ksp_monitor_true_residual \ >> -fieldsplit_velocity_ksp_type preonly \ >> -fieldsplit_velocity_pc_type lu \ >> -fieldsplit_velocity_pc_factor_mat_solver_package mumps \ >> -fieldsplit_pressure_ksp_type gmres \ >> -fieldsplit_pressure_ksp_max_it 10 \ >> -fieldsplit_pressure_ksp_rtol 1.0e-2 \ >> -fieldsplit_pressure_pc_type lsc \ >> >> This makes no sense unless you specify an auxiliary operator. Just leave it at jacobi. When you use LU >> for velocity, it will converge in 1 iterate. Since it doesn't, it means you have a problem with the null space. > So even for the driven cavity, the null space is set, and again I use the same solver, it needs something like 10 or 15 iterations. I don't understand your argument, that using LU for the velocity part, this solver should converge in 1 iterations. It still has an inexact solver for the Schur complement and the solver for the composite matrix in LSC is also inexact. > > Thomas > >> >> Matt >> >> -fieldsplit_pressure_lsc_ksp_type bcgs \ >> -fieldsplit_pressure_lsc_ksp_max_it 10 \ >> -fieldsplit_pressure_lsc_ksp_rtol 1.0e-2 \ >> -fieldsplit_pressure_lsc_pc_type hypre >> >> The direct solver for the velocity part is just for debugging and should be replaced when everything else works fine. The point is, that I found this constellation to be not very efficient. It takes around 20 to 30 iterations, which takes around 30 seconds on a very small problem size (around 20000 global unknows for each velocity component and 5000 global unknowns for the pressure) on a very fast desktop CPU (some new Intel Xeno with 4 core). Any hints for improvements? >> >> Thomas >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > From w_ang_temp at 163.com Thu Jun 14 22:18:30 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Fri, 15 Jun 2012 11:18:30 +0800 (CST) Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> <7fbcf7a5.9794.137d594d682.Coremail.w_ang_temp@163.com> Message-ID: <2f4470ff.63c6.137ee263477.Coremail.w_ang_temp@163.com> Hello, Matt After my careful testing, I get some findings. Firstly, considering the approach of setting one row at a time. I change my codes from setting one data per time to one row per time (codes attached). I find that both of the times are almost equal and cannot be ignored . In my opinion, the latter should take less time than the former. the codes have been proved to have the right results and I think my testing datas are enough to show that they almost have the same time for setting the matrix. Secondly, as you said in a previous post('CSR Sparse Matrix Query', 2007), there is no Fortran interface for the function 'MatCreateMPIAIJWithArrays'. I am not sure if it is true now because the docs do not specify this and I connot compile with it atfer a simple test. In my opinion, you prefer to do the thing of setting values from CSR format using the way of setting values with MatSetValus for each row rather than using MatCreateMPIAIJWithArrays. am I right? Thanks again. Jim -----coding:per row(you can compare it with the first page int this post of one data per time )----- !NROWIN,NNZIJ,values:the CSR variables of the main function !NROWIN,NNZIJ:one-based,so get zero-based variables rowIndex,columns do 10,II=Istart+1,Iend !no-zero numbers of this row rowNum=NROWIN(II+1)-NROWIN(II) allocate(nColPerRow(rowNum)) allocate(valuePerRow(rowNum)) kValStart=NROWIN(II)+1-1 kValEnd=NROWIN(II)+rowNum-1 !column index nColPerRow=NNZIJ(kValStart:kValEnd)-1 valuePerRow=values(kValStart:kValEnd) nRow=II-1 call MatSetValues(A,ione,nRow,rowNum,nColPerRow,valuePerRow, & INSERT_VALUES,ierr) deallocate(nColPerRow) deallocate(valuePerRow) 10 continue ---------------------coding----------------------------- >? 2012-06-10 20:49:13?"Matthew Knepley" ??? >On Sun, Jun 10, 2012 at 4:48 PM, w_ang_temp wrote: >Hello, Barry > I try to use MatCreateSeqAIJWithArrays to deal with my problem. Since >there is no example in the doc, I find some examples in the web and use it. >And I get some errors. > When the processor is 1(mpiexec -n 1 ./ex4f), the results are ok. While >it is more than one, I get the errors. So I think I miss some piece of >information about MatCreateSeqAIJWithArrays. >1) Check the error codes. Always! > Thanks. > Jim >----------------------------code------------------------------------------- > call MatCreate(PETSC_COMM_WORLD,A,ierr) > call MatSetType(A,MATAIJ,ierr) > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr) > call MatSetFromOptions(A,ierr) > call MatGetOwnershipRange(A,Istart,Iend,ierr) > !NROWIN,NNZIJ,values:the CSR variables of the main function > !NROWIN,NNZIJ:one-based,so get zero-based variables rowIndex,columns > allocate(rowIndex(JFLNG+1)) > allocate(columns(MAXNNZ)) > rowIndex=NROWIN-1 > columns=NNZIJ-1 > call MatCreateSeqAIJWithArrays(MPI_COMM_WORLD,m,n,rowIndex,columns, > & values,A, ierr) >2) This can only be called in serial. In parallel, you need MatCreateMPIAIJWithArrays() >3) I still believe you are really doing the wrong thing. The right solution is to set one row at > a time. I will bet dollars to doughnuts that the assembly time is not noticeable in your program. > Matt > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >----------------------------code------------------------------------------- >----------------------------Error Message---------------------------------- >......... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajay.rawat83 at gmail.com Fri Jun 15 05:17:40 2012 From: ajay.rawat83 at gmail.com (Ajay Rawat) Date: Fri, 15 Jun 2012 15:47:40 +0530 Subject: [petsc-users] Broken link for FFC Message-ID: Dear Dev While configuring the PETSc with --download-ffc options it throws an error Downloaded package FFC from: http://www.fenics.org/pub/software/ffc/v0.3/ffc-0.3.3.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * Alternatively, you can download the above URL manually, to /yourselectedlocation/ffc-0.3.3.tar.gz and use the configure option: --download-ffc=/yourselectedlocation/ffc-0.3.3.tar.gz I guess the link is not correct. I have one more doubt regarding ffc. I see no examples regarding how to use ffc inside PETSc, or it is used inside fenics. With regards, Ajay Rawat Kalpakkam, IGCAR ------------------------------------------------------------------------- Save Himalayas.... ------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 15 06:03:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 15 Jun 2012 06:03:43 -0500 Subject: [petsc-users] Broken link for FFC In-Reply-To: References: Message-ID: On Fri, Jun 15, 2012 at 5:17 AM, Ajay Rawat wrote: > Dear Dev > > While configuring the PETSc with --download-ffc options it throws an error > > Downloaded package FFC from: > http://www.fenics.org/pub/software/ffc/v0.3/ffc-0.3.3.tar.gz is not a > tarball. > [or installed python cannot process compressed files] > This server has not existed for a while (a troll threatened legal action over the name, IIRC). Matt will have to check whether new versions of FFC work with his code. It's not used by "mainstream" PETSc. > * If you are behind a firewall - please fix your proxy and rerun > ./configure > For example at LANL you may need to set the environmental variable > http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov > * Alternatively, you can download the above URL manually, to > /yourselectedlocation/ffc-0.3.3.tar.gz > and use the configure option: > --download-ffc=/yourselectedlocation/ffc-0.3.3.tar.gz > > I guess the link is not correct. > > I have one more doubt regarding ffc. I see no examples regarding how to > use ffc inside PETSc, or it is used inside fenics. > > With regards, > > Ajay Rawat > Kalpakkam, IGCAR > > ------------------------------------------------------------------------- > Save Himalayas.... > ------------------------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 15 06:25:24 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 19:25:24 +0800 Subject: [petsc-users] From CSR format to MatSetValues In-Reply-To: <2f4470ff.63c6.137ee263477.Coremail.w_ang_temp@163.com> References: <12d01be.1b71a.137caca40d5.Coremail.w_ang_temp@163.com> <23BB90AE-790B-440B-944B-620E262E2140@mcs.anl.gov> <7fbcf7a5.9794.137d594d682.Coremail.w_ang_temp@163.com> <2f4470ff.63c6.137ee263477.Coremail.w_ang_temp@163.com> Message-ID: On Fri, Jun 15, 2012 at 11:18 AM, w_ang_temp wrote: > Hello, Matt > > After my careful testing, I get some findings. > > Firstly, considering the approach of setting one row at a time. I > change my codes > from setting one data per time to one row per time (codes attached). I > find that both > of the times are almost equal and cannot be ignored . In my opinion, the > latter should > take less time than the former. the codes have been proved to have the > right results > and I think my testing datas are enough to show that they almost have the > same time > for setting the matrix. > I do not believe you analysis. Send the output of -log_summary -info to petsc-maint at mcs.anl.gov Matt > > Secondly, as you said in a previous post('CSR Sparse Matrix Query', > 2007), there > is no Fortran interface for the function 'MatCreateMPIAIJWithArrays'. I am > not sure > if it is true now because the docs do not specify this and I connot > compile with > it atfer a simple test. In my opinion, you prefer to do the thing of > setting values from > CSR format using the way of setting values with MatSetValus for each row > rather than > using MatCreateMPIAIJWithArrays. am I right? > > Thanks again. > Jim > -----coding:per row(you can compare it with the first page int this post > of one data per time )----- > > !NROWIN,NNZIJ,values:the CSR variables of the main function > !NROWIN,NNZIJ:one-based,so get zero-based variables rowIndex,columns > do 10,II=Istart+1,Iend > !no-zero numbers of this row > rowNum=NROWIN(II+1)-NROWIN(II) > > allocate(nColPerRow(rowNum)) > allocate(valuePerRow(rowNum)) > > kValStart=NROWIN(II)+1-1 > kValEnd=NROWIN(II)+rowNum-1 > > !column index > nColPerRow=NNZIJ(kValStart:kValEnd)-1 > valuePerRow=values(kValStart:kValEnd) > nRow=II-1 > call MatSetValues(A,ione,nRow,rowNum,nColPerRow,valuePerRow, > & INSERT_VALUES,ierr) > deallocate(nColPerRow) > deallocate(valuePerRow) > 10 continue > ---------------------coding----------------------------- > > > >? 2012-06-10 20:49:13?"Matthew Knepley" ??? > > >On Sun, Jun 10, 2012 at 4:48 PM, w_ang_temp wrote: > >> >Hello, Barry >> >> > I try to use MatCreateSeqAIJWithArrays to deal with my problem. >> Since >> >there is no example in the doc, I find some examples in the web and use >> it. >> >And I get some errors. >> >> > When the processor is 1(mpiexec -n 1 ./ex4f), the results are ok. >> While >> >it is more than one, I get the errors. So I think I miss some piece of >> >information about MatCreateSeqAIJWithArrays. >> > > >1) Check the error codes. Always! > > >> > Thanks. >> > Jim >> >> >> >----------------------------code------------------------------------------- >> > call MatCreate(PETSC_COMM_WORLD,A,ierr) >> > call MatSetType(A,MATAIJ,ierr) >> > call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m,n,ierr) >> > call MatSetFromOptions(A,ierr) >> > call MatGetOwnershipRange(A,Istart,Iend,ierr) >> > > >> > !NROWIN,NNZIJ,values:the CSR variables of the main function >> > !NROWIN,NNZIJ:one-based,so get zero-based variables >> rowIndex,columns >> > allocate(rowIndex(JFLNG+1)) >> > allocate(columns(MAXNNZ)) >> > rowIndex=NROWIN-1 >> > columns=NNZIJ-1 >> >> > call MatCreateSeqAIJWithArrays(MPI_COMM_WORLD,m,n,rowIndex,columns, >> > & values,A, ierr) >> > > >2) This can only be called in serial. In parallel, you need > MatCreateMPIAIJWithArrays() > > >3) I still believe you are really doing the wrong thing. The right > solution is to set one row at > > a time. I will bet dollars to doughnuts that the assembly time is not > noticeable in your program. > > > Matt > > >> >> > call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >> > call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >> >> >----------------------------code------------------------------------------- >> >> >----------------------------Error >> Message---------------------------------- >> >......... >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 15 06:29:58 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 19:29:58 +0800 Subject: [petsc-users] Broken link for FFC In-Reply-To: References: Message-ID: On Fri, Jun 15, 2012 at 7:03 PM, Jed Brown wrote: > On Fri, Jun 15, 2012 at 5:17 AM, Ajay Rawat wrote: > >> Dear Dev >> >> While configuring the PETSc with --download-ffc options it throws an error >> >> Downloaded package FFC from: >> http://www.fenics.org/pub/software/ffc/v0.3/ffc-0.3.3.tar.gz is not a >> tarball. >> [or installed python cannot process compressed files] >> > > This server has not existed for a while (a troll threatened legal action > over the name, IIRC). Matt will have to check whether new versions of FFC > work with his code. It's not used by "mainstream" PETSc. > Yes, I will have to update this link. The reason this package exists is that I was using it to generate CUDA code in a restricted case. We now have a more general routine however, so it is not being used. Matt > > >> * If you are behind a firewall - please fix your proxy and rerun >> ./configure >> For example at LANL you may need to set the environmental variable >> http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov >> * Alternatively, you can download the above URL manually, to >> /yourselectedlocation/ffc-0.3.3.tar.gz >> and use the configure option: >> --download-ffc=/yourselectedlocation/ffc-0.3.3.tar.gz >> >> I guess the link is not correct. >> >> I have one more doubt regarding ffc. I see no examples regarding how to >> use ffc inside PETSc, or it is used inside fenics. >> >> With regards, >> >> Ajay Rawat >> Kalpakkam, IGCAR >> >> ------------------------------------------------------------------------- >> Save Himalayas.... >> ------------------------------------------------------------------------- >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From andy.terrel at gmail.com Fri Jun 15 06:33:01 2012 From: andy.terrel at gmail.com (Andy Ray Terrel) Date: Fri, 15 Jun 2012 13:33:01 +0200 Subject: [petsc-users] Broken link for FFC In-Reply-To: References: Message-ID: Just change www.fenics.org to www.fenicsproject.org, but 0.3.3 is terribly old. -- Andy On Fri, Jun 15, 2012 at 1:29 PM, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 7:03 PM, Jed Brown wrote: > >> On Fri, Jun 15, 2012 at 5:17 AM, Ajay Rawat wrote: >> >>> Dear Dev >>> >>> While configuring the PETSc with --download-ffc options it throws an >>> error >>> >>> Downloaded package FFC from: >>> http://www.fenics.org/pub/software/ffc/v0.3/ffc-0.3.3.tar.gz is not a >>> tarball. >>> [or installed python cannot process compressed files] >>> >> >> This server has not existed for a while (a troll threatened legal action >> over the name, IIRC). Matt will have to check whether new versions of FFC >> work with his code. It's not used by "mainstream" PETSc. >> > > Yes, I will have to update this link. The reason this package exists is > that I was using it to generate CUDA > code in a restricted case. We now have a more general routine however, so > it is not being used. > > Matt > > >> >> >>> * If you are behind a firewall - please fix your proxy and rerun >>> ./configure >>> For example at LANL you may need to set the environmental variable >>> http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov >>> * Alternatively, you can download the above URL manually, to >>> /yourselectedlocation/ffc-0.3.3.tar.gz >>> and use the configure option: >>> --download-ffc=/yourselectedlocation/ffc-0.3.3.tar.gz >>> >>> I guess the link is not correct. >>> >>> I have one more doubt regarding ffc. I see no examples regarding how to >>> use ffc inside PETSc, or it is used inside fenics. >>> >>> With regards, >>> >>> Ajay Rawat >>> Kalpakkam, IGCAR >>> >>> ------------------------------------------------------------------------- >>> Save Himalayas.... >>> ------------------------------------------------------------------------- >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 15 07:22:49 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 15 Jun 2012 14:22:49 +0200 Subject: [petsc-users] -log_summary for MatMult Message-ID: <4FDB2919.9040403@gfz-potsdam.de> Hello, When I call MatMult with MPIDENSE matrix in log summary I see: Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ MatMult 1 1.0 9.5740e-01 1.5 8.19e+08 1.0 0.0e+00 0.0e+00 1.0e+00 2 40 0 0 5 2 40 0 0 6 3419 MatMultTranspose 1 1.0 1.0793e+00 1.4 8.18e+08 1.0 0.0e+00 0.0e+00 2.0e+00 2 40 0 0 11 2 40 0 0 11 3030 Why do number of messages and their length equal zero? -- Regards, Alexander From knepley at gmail.com Fri Jun 15 07:27:20 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 20:27:20 +0800 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: <4FDB2919.9040403@gfz-potsdam.de> References: <4FDB2919.9040403@gfz-potsdam.de> Message-ID: On Fri, Jun 15, 2012 at 8:22 PM, Alexander Grayver wrote: > Hello, > > When I call MatMult with MPIDENSE matrix in log summary I see: > > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > ------------------------------**------------------------------** > ------------------------------**------------------------------ > MatMult 1 1.0 9.5740e-01 1.5 8.19e+08 1.0 0.0e+00 0.0e+00 > 1.0e+00 2 40 0 0 5 2 40 0 0 6 3419 > MatMultTranspose 1 1.0 1.0793e+00 1.4 8.18e+08 1.0 0.0e+00 0.0e+00 > 2.0e+00 2 40 0 0 11 2 40 0 0 11 3030 > > > Why do number of messages and their length equal zero? Because it uses only collectives. MAtt > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 15 07:31:37 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 15 Jun 2012 14:31:37 +0200 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: References: <4FDB2919.9040403@gfz-potsdam.de> Message-ID: <4FDB2B29.60400@gfz-potsdam.de> Matt, According to that code: 486:*PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)* 487:{ 488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD); 493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD); 494: MatMult_SeqDense(mdn->A,mdn->lvec,yy); 495: return(0); 496:} Each process has its own local copy of the whole vector vectors? On 15.06.2012 14:27, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 8:22 PM, Alexander Grayver > > wrote: > > Hello, > > When I call MatMult with MPIDENSE matrix in log summary I see: > > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess > Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > MatMult 1 1.0 9.5740e-01 1.5 8.19e+08 1.0 0.0e+00 > 0.0e+00 1.0e+00 2 40 0 0 5 2 40 0 0 6 3419 > MatMultTranspose 1 1.0 1.0793e+00 1.4 8.18e+08 1.0 0.0e+00 > 0.0e+00 2.0e+00 2 40 0 0 11 2 40 0 0 11 3030 > > > Why do number of messages and their length equal zero? > > > Because it uses only collectives. > > MAtt > > > -- > Regards, > Alexander > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 15 07:46:50 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 20:46:50 +0800 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: <4FDB2B29.60400@gfz-potsdam.de> References: <4FDB2919.9040403@gfz-potsdam.de> <4FDB2B29.60400@gfz-potsdam.de> Message-ID: On Fri, Jun 15, 2012 at 8:31 PM, Alexander Grayver wrote: > ** > Matt, > > According to that code: > > 486: *PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)*487: {488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; > 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);494: MatMult_SeqDense(mdn->A,mdn->lvec,yy);495: return(0);496: } > > > Each process has its own local copy of the whole vector vectors? > I am not sure what your point is. VecScatter is just an interface that has many implementations. What I am telling you is that the timing you sent only does reductions, no p2p messages. Matt > On 15.06.2012 14:27, Matthew Knepley wrote: > > On Fri, Jun 15, 2012 at 8:22 PM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> Hello, >> >> When I call MatMult with MPIDENSE matrix in log summary I see: >> >> Event Count Time (sec) Flops >> --- Global --- --- Stage --- Total >> Max Ratio Max Ratio Max Ratio Mess Avg len >> Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s >> >> ------------------------------------------------------------------------------------------------------------------------ >> MatMult 1 1.0 9.5740e-01 1.5 8.19e+08 1.0 0.0e+00 0.0e+00 >> 1.0e+00 2 40 0 0 5 2 40 0 0 6 3419 >> MatMultTranspose 1 1.0 1.0793e+00 1.4 8.18e+08 1.0 0.0e+00 0.0e+00 >> 2.0e+00 2 40 0 0 11 2 40 0 0 11 3030 >> >> >> Why do number of messages and their length equal zero? > > > Because it uses only collectives. > > MAtt > > >> >> -- >> Regards, >> Alexander >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Fri Jun 15 08:18:27 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 15 Jun 2012 15:18:27 +0200 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: References: <4FDB2919.9040403@gfz-potsdam.de> <4FDB2B29.60400@gfz-potsdam.de> Message-ID: <4FDB3623.8060408@gfz-potsdam.de> On 15.06.2012 14:46, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 8:31 PM, Alexander Grayver > > wrote: > > Matt, > > According to that code: > > 486:*PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)* > 487:{ > 488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; > > 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD); > 493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD); > 494: MatMult_SeqDense(mdn->A,mdn->lvec,yy); > 495: return(0); > 496:} > > > Each process has its own local copy of the vector? > > > I am not sure what your point is. VecScatter is just an interface that > has many implementations. I'm trying to estimate the amount of data needs to be communicated over all processes during this operation. In debugger I see VecScatter from the code above reduces to the MPI_Allgatherv and results in (assuming vector is distributed uniformly) bytes_send_received = num_of_proc * ((num_of_proc - 1) * vec_size_local) * 2 * sizeof(PetscScalar) Does that look reasonable? Thanks. -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 15 09:14:28 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 15 Jun 2012 22:14:28 +0800 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: <4FDB3623.8060408@gfz-potsdam.de> References: <4FDB2919.9040403@gfz-potsdam.de> <4FDB2B29.60400@gfz-potsdam.de> <4FDB3623.8060408@gfz-potsdam.de> Message-ID: On Fri, Jun 15, 2012 at 9:18 PM, Alexander Grayver wrote: > ** > On 15.06.2012 14:46, Matthew Knepley wrote: > > On Fri, Jun 15, 2012 at 8:31 PM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> Matt, >> >> According to that code: >> >> 486: *PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)*487: {488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; >> 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);494: MatMult_SeqDense(mdn->A,mdn->lvec,yy);495: return(0);496: } >> >> >> Each process has its own local copy of the vector? >> > > I am not sure what your point is. VecScatter is just an interface that > has many implementations. > > > I'm trying to estimate the amount of data needs to be communicated over > all processes during this operation. > In debugger I see VecScatter from the code above reduces to the > MPI_Allgatherv and results in (assuming vector is distributed uniformly) > > bytes_send_received = num_of_proc * ((num_of_proc - 1) * vec_size_local) * > 2 * sizeof(PetscScalar) > > Does that look reasonable? > This is not really a useful exercise, since a) PETSc does not currently have an optimized parallel dense implementation b) We are implementing an Elemental interface this summer. You can try it out in petsc-dev c) Elemental is much more efficient than our simple implementation, and uses a unique approach to communication (all reductions) I would take the comp+comm estimates from Jack's slides on Elemental Matt > Thanks. > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Fri Jun 15 09:51:46 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Fri, 15 Jun 2012 09:51:46 -0500 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: References: <4FDB2919.9040403@gfz-potsdam.de> <4FDB2B29.60400@gfz-potsdam.de> <4FDB3623.8060408@gfz-potsdam.de> Message-ID: On Fri, Jun 15, 2012 at 9:14 AM, Matthew Knepley wrote: > On Fri, Jun 15, 2012 at 9:18 PM, Alexander Grayver < > agrayver at gfz-potsdam.de> wrote: > >> ** >> On 15.06.2012 14:46, Matthew Knepley wrote: >> >> On Fri, Jun 15, 2012 at 8:31 PM, Alexander Grayver < >> agrayver at gfz-potsdam.de> wrote: >> >>> Matt, >>> >>> According to that code: >>> >>> 486: *PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)*487: {488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; >>> 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);494: MatMult_SeqDense(mdn->A,mdn->lvec,yy);495: return(0);496: } >>> >>> >>> Each process has its own local copy of the vector? >>> >> >> I am not sure what your point is. VecScatter is just an interface that >> has many implementations. >> >> >> I'm trying to estimate the amount of data needs to be communicated over >> all processes during this operation. >> In debugger I see VecScatter from the code above reduces to the >> MPI_Allgatherv and results in (assuming vector is distributed uniformly) >> >> bytes_send_received = num_of_proc * ((num_of_proc - 1) * vec_size_local) >> * 2 * sizeof(PetscScalar) >> >> Does that look reasonable? >> > > This is not really a useful exercise, since > > a) PETSc does not currently have an optimized parallel dense > implementation > > b) We are implementing an Elemental interface this summer. You can try > it out in petsc-dev > > c) Elemental is much more efficient than our simple implementation, and > uses a unique > approach to communication (all reductions) > > I would take the comp+comm estimates from Jack's slides on Elemental > > Matt > > Jed and I just discussed this an hour or so ago; the MatMult (Gemv in BLAS parlance) implementation via Elemental will, in the simplest case of a square n x n matrix distributed over a square sqrt(p) x sqrt(p) process grid, require: 1) A gather (via a padded MPI_Gather, not MPI_Gatherv) within teams of sqrt(p) processes, with a per-process communication volume of approximately n/sqrt(p), and a latency cost of log2(sqrt(p)) ~= 1/2 log2(p) 2) A Gemv with the local part of A against the just gathered part of x, (z[MC,*] := A[MC,MR] x[MR,* ] in Elemental notation) 3) A sum-scatter within teams of sqrt(p) processes (via MPI_Reduce_scatter, or MPI_Reduce_scatter_block if it is available), requiring roughly n/sqrt(p) per-process communication volume and roughly log2(sqrt(p)) latency again. Thus, the total per-process communication volume will be about 2 n / sqrt(p), the total latency will be about log2(p), and the local computation is roughly the optimal value, n^2 / p. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From jack.poulson at gmail.com Fri Jun 15 09:58:45 2012 From: jack.poulson at gmail.com (Jack Poulson) Date: Fri, 15 Jun 2012 09:58:45 -0500 Subject: [petsc-users] -log_summary for MatMult In-Reply-To: References: <4FDB2919.9040403@gfz-potsdam.de> <4FDB2B29.60400@gfz-potsdam.de> <4FDB3623.8060408@gfz-potsdam.de> Message-ID: On Fri, Jun 15, 2012 at 9:51 AM, Jack Poulson wrote: > On Fri, Jun 15, 2012 at 9:14 AM, Matthew Knepley wrote: > >> On Fri, Jun 15, 2012 at 9:18 PM, Alexander Grayver < >> agrayver at gfz-potsdam.de> wrote: >> >>> ** >>> On 15.06.2012 14:46, Matthew Knepley wrote: >>> >>> On Fri, Jun 15, 2012 at 8:31 PM, Alexander Grayver < >>> agrayver at gfz-potsdam.de> wrote: >>> >>>> Matt, >>>> >>>> According to that code: >>>> >>>> 486: *PetscErrorCode MatMult_MPIDense(Mat mat,Vec xx,Vec yy)*487: {488: Mat_MPIDense *mdn = (Mat_MPIDense*)mat->data; >>>> 492: VecScatterBegin(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);493: VecScatterEnd(mdn->Mvctx,xx,mdn->lvec,INSERT_VALUES,SCATTER_FORWARD);494: MatMult_SeqDense(mdn->A,mdn->lvec,yy);495: return(0);496: } >>>> >>>> >>>> Each process has its own local copy of the vector? >>>> >>> >>> I am not sure what your point is. VecScatter is just an interface that >>> has many implementations. >>> >>> >>> I'm trying to estimate the amount of data needs to be communicated over >>> all processes during this operation. >>> In debugger I see VecScatter from the code above reduces to the >>> MPI_Allgatherv and results in (assuming vector is distributed uniformly) >>> >>> bytes_send_received = num_of_proc * ((num_of_proc - 1) * vec_size_local) >>> * 2 * sizeof(PetscScalar) >>> >>> Does that look reasonable? >>> >> >> This is not really a useful exercise, since >> >> a) PETSc does not currently have an optimized parallel dense >> implementation >> >> b) We are implementing an Elemental interface this summer. You can try >> it out in petsc-dev >> >> c) Elemental is much more efficient than our simple implementation, and >> uses a unique >> approach to communication (all reductions) >> >> I would take the comp+comm estimates from Jack's slides on Elemental >> >> Matt >> >> > > Jed and I just discussed this an hour or so ago; the MatMult (Gemv in BLAS > parlance) implementation via Elemental will, in the simplest case of a > square n x n matrix distributed over a square sqrt(p) x sqrt(p) process > grid, require: > > 1) A gather (via a padded MPI_Gather, not MPI_Gatherv) within teams of > sqrt(p) processes, with a per-process communication volume of approximately > n/sqrt(p), and a latency cost of log2(sqrt(p)) ~= 1/2 log2(p) > 2) A Gemv with the local part of A against the just gathered part of x, > (z[MC,*] := A[MC,MR] x[MR,* ] in Elemental notation) > 3) A sum-scatter within teams of sqrt(p) processes (via > MPI_Reduce_scatter, or MPI_Reduce_scatter_block if it is available), > requiring roughly n/sqrt(p) per-process communication volume and roughly > log2(sqrt(p)) latency again. > > Thus, the total per-process communication volume will be about 2 n / > sqrt(p), the total latency will be about log2(p), and the local computation > is roughly the optimal value, n^2 / p. > > Jack > I just noticed two minor corrections to what I just said: 1) The version that will be used by PETSc will actually use a padded MPI_Allgather, not a padded MPI_Gather, but the cost is essentially the same. 2) The sequential cost of a square matrix-vector multiplication is 2 n^2, so the local computation will be roughly 2 n^2 / p flops. Jack -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Jun 15 14:23:49 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 15 Jun 2012 20:23:49 +0100 (BST) Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> Message-ID: <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> Dear Jed, Thanks for attention! I recall my previous thread; copy and paste here, and then I try to ask: Suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a post-smoother S, application of the preconditioner to the vector "b" would be C b + S (b - A C b). If we drop the post-smoother by setting S = I, We get Prec = C b + b - A C b. Note that there is no presmoother. This is the preconditioner I have used before, in PCMG framework, with no pre-smoothing and Richardson(without any preconditioner) as post smoother.? I need to apply the updated version of above Prec = C b + (b - A C b), where A is replaced by M^-1 A , and A_H by M_H^-1 A_H . Now does it make sense : 1)? incorporating M as only preconditioner in presmoother will replace A by M^-1 A in Prec = C b + (b - A C b) ? 2) incorporating M as preconditioner to CG-solver will replace A_H by (M_H^-1 A_H ) ? I am kinda of sure about this point 2. Thanks, Abdul From: Jed Brown To: Abdul Hanan Sheikh >Cc: PETSc users list >Sent: Wednesday, 13 June 2012, 18:39 >Subject: Re: [petsc-users] How to implement projection preconditioner? > > >On Wed, Jun 13, 2012 at 10:54 AM, Abdul Hanan Sheikh wrote: > >Thanks for reply! >> >>Well at first look, it looks like that M can be put as precondittioning operator for smoother >>where keeping ksp as preonly in pre-smoother context.? >>I am not sure, this leads to the preconditioner desired, coz putting M as preconditioner in >> >>pre-smoother alters the residual passed to coarse-grid correction,? i.e. >> >>CGC corrects the pre-smoothed solution. >> > > >That's kind of the point of pre-smoothing. Can you explain more precisely (or just try it). >? > >> >> >> >> >> >>>________________________________ >>> From: Jed Brown >>>To: Abdul Hanan Sheikh ; PETSc users list >>>Cc: Abdul - CITG >>>Sent: Friday, 8 June 2012, 19:01 >>>Subject: Re: [petsc-users] How to implement projection preconditioner? >>> >>> >>> >>>Isn't this just putting M and M_H as the preconditioning operator for the smoother? >>> >>> >>>On Fri, Jun 8, 2012 at 11:58 AM, Abdul Hanan Sheikh wrote: >>> >>>Dear all, >>>> >>>>Summer greetings, >>>> >>>>I am back with an other query. >>>>Before I successfully implement the projection preconditioner which was simply >>>> >>>>the coarse grid correction operator P = I - A*(P*A_H*R); >>>> >>>>I implemented this simply keeping both pre and post smoothing dummy in PCMG setup. >>>> >>>>Now I need to revise this and re-implement this where I replace A and coarse operator A_H with >>>> >>>>preconditioned one i.e. M^-1 A and M^-1 A_H respectively. Thus new projection reads as >>>> >>>> >>>> >>>> >>>>P_new = I - (M^-1 A) {P*(M_H^-1 A_H)*R} >>>> >>>> >>>>Any suggestion to implement this in Petsc would be appreciated. >>>> >>>> >>>> >>>>Thanking in anticipation. >>>> >>>>with regards, Abdul >>>> >>> >>> >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 15 14:36:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 15 Jun 2012 14:36:23 -0500 Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> Message-ID: On Fri, Jun 15, 2012 at 2:23 PM, Abdul Hanan Sheikh wrote: > I recall my previous thread; copy and paste here, and then I try to ask: > > Suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a > post-smoother S, application of the preconditioner to the vector "b" would > be C b + S (b - A C b). If we drop the post-smoother by setting S = I, We > get Prec = C b + b - A C b. Note that there is no presmoother. > > This is the preconditioner I have used before, in PCMG framework, with no > pre-smoothing and Richardson(without any preconditioner) as post smoother. > > I need to apply the updated version of above Prec = C b + (b - A C b), > where A is replaced by M^-1 A , and A_H by M_H^-1 A_H . > With left preconditioning, you also have to apply the preconditioner to the right hand side, thus making it look just like a smoother. > > Now does it make sense : > > 1) incorporating M as only preconditioner in presmoother will replace A > by M^-1 A in Prec = C b + (b - A C b) ? > No, it uses the preconditioner the way that preconditioners work, see above. > 2) incorporating M as preconditioner to CG-solver will replace A_H by > (M_H^-1 A_H ) ? I am kinda of sure about this point 2. > C already contains A_H^{-1}. Replacing that by M_H^{-1} is an approximate coarse level solve (that's okay and seems to be what you want, just not what you have written, which is not a consistent method). -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Jun 15 15:07:38 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 15 Jun 2012 21:07:38 +0100 (BST) Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> Message-ID: <1339790858.47311.YahooMailNeo@web28703.mail.ir2.yahoo.com> Thank you for reply! I didnt get , what does it mean by applying the preconditioner to the right hand side, making it look just like a smoother ? More simply, how can I implement Prec = \bar C b + (b - \bar A \bar C b) where \bar C =? P (M_H^-1 A_H)^-1 R and \bar A = M^-1 A. Note: we know how to implement Prec = C b + (b - A C b). Many thanks, Abdul >________________________________ > From: Jed Brown >To: Abdul Hanan Sheikh >Cc: PETSc users list >Sent: Friday, 15 June 2012, 21:36 >Subject: Re: [petsc-users] How to implement projection preconditioner? > > >On Fri, Jun 15, 2012 at 2:23 PM, Abdul Hanan Sheikh wrote: > >I recall my previous thread; copy and paste here, and then I try to ask: >> >> >> >>Suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a post-smoother S, application of the preconditioner to the vector "b" would be C b + S (b - A C b). If we drop the post-smoother by setting S = I, We get Prec = C b + b - A C b. Note that there is no presmoother. >> >> >> >>This is the preconditioner I have used before, in PCMG framework, with no pre-smoothing and Richardson(without any preconditioner) as post smoother.? >> >> >>I need to apply the updated version of above Prec = C b + (b - A C b), where A is replaced by M^-1 A , and A_H by M_H^-1 A_H . >> > > >With left preconditioning, you also have to apply the preconditioner to the right hand side, thus making it look just like a smoother. >? > >>Now does it make sense : >> >>1)? incorporating M as only preconditioner in presmoother will replace A by M^-1 A in Prec = C b + (b - A C b) ? >> > > >No, it uses the preconditioner the way that preconditioners work, see above. >? >2) incorporating M as preconditioner to CG-solver will replace A_H by (M_H^-1 A_H ) ? I am kinda of sure about this point 2. > >C already contains A_H^{-1}. Replacing that by M_H^{-1} is an approximate coarse level solve (that's okay and seems to be what you want, just not what you have written, which is not a consistent method). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 15 16:58:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 15 Jun 2012 16:58:58 -0500 Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: <1339790858.47311.YahooMailNeo@web28703.mail.ir2.yahoo.com> References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> <1339790858.47311.YahooMailNeo@web28703.mail.ir2.yahoo.com> Message-ID: Let's get even simpler, suppose I start with A x = b The left-preconditioned form of this system is *not* M^{-1} A x = b it is M^{-1} A x = M^{-1} b Run that through your analysis and you'll find that what your asking for doesn't make sense. It often helps keep things straight if you consider that the residual and solution do not lie in the same space, in which case "no preconditioning" is the "trivial" map from the residual space back to the state space. On Fri, Jun 15, 2012 at 3:07 PM, Abdul Hanan Sheikh wrote: > Thank you for reply! > I didnt get , what does it mean by applying the preconditioner to the > right hand side, making it look just like a smoother ? > > More simply, how can I implement Prec = \bar C b + (b - \bar A \bar C b) > where \bar C = P (M_H^-1 A_H)^-1 R > and \bar A = M^-1 A. > Note: we know how to implement Prec = C b + (b - A C b). > > Many thanks, Abdul > > ------------------------------ > *From:* Jed Brown > *To:* Abdul Hanan Sheikh > *Cc:* PETSc users list > *Sent:* Friday, 15 June 2012, 21:36 > > *Subject:* Re: [petsc-users] How to implement projection preconditioner? > > On Fri, Jun 15, 2012 at 2:23 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > I recall my previous thread; copy and paste here, and then I try to ask: > > Suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a > post-smoother S, application of the preconditioner to the vector "b" would > be C b + S (b - A C b). If we drop the post-smoother by setting S = I, We > get Prec = C b + b - A C b. Note that there is no presmoother. > > This is the preconditioner I have used before, in PCMG framework, with no > pre-smoothing and Richardson(without any preconditioner) as post smoother. > > I need to apply the updated version of above Prec = C b + (b - A C b), > where A is replaced by M^-1 A , and A_H by M_H^-1 A_H . > > > With left preconditioning, you also have to apply the preconditioner to > the right hand side, thus making it look just like a smoother. > > > > Now does it make sense : > > 1) incorporating M as only preconditioner in presmoother will replace A > by M^-1 A in Prec = C b + (b - A C b) ? > > > No, it uses the preconditioner the way that preconditioners work, see > above. > > > 2) incorporating M as preconditioner to CG-solver will replace A_H by > (M_H^-1 A_H ) ? I am kinda of sure about this point 2. > > > C already contains A_H^{-1}. Replacing that by M_H^{-1} is an approximate > coarse level solve (that's okay and seems to be what you want, just not > what you have written, which is not a consistent method). > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jun 15 19:58:17 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 15 Jun 2012 20:58:17 -0400 Subject: [petsc-users] Obtaining more ghost points at EE, SS etc using DM Message-ID: <4FDBDA29.7020809@gmail.com> Hi, I'm trying to implement DM in my code. So far, I found that it is very convenient to use its features. I am currently using DMDA STENCIL STAR. However, in some parts of my code, I need to update ghost points at east east, west west etc. DMGlobalToLocalBegin/end can only update east, west etc. What's the easiest way to update these additional points? Do I have to use MPI_Send? Also, how is the memory allocated? I think the east east values will be outside the local vector, is that so? Thank you. -- Yours sincerely, TAY wee-beng From jedbrown at mcs.anl.gov Fri Jun 15 20:04:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 15 Jun 2012 20:04:58 -0500 Subject: [petsc-users] Obtaining more ghost points at EE, SS etc using DM In-Reply-To: <4FDBDA29.7020809@gmail.com> References: <4FDBDA29.7020809@gmail.com> Message-ID: If you need corner points, use DMDA_STENCIL_BOX. On Fri, Jun 15, 2012 at 7:58 PM, TAY wee-beng wrote: > Hi, > > I'm trying to implement DM in my code. So far, I found that it is very > convenient to use its features. I am currently using DMDA STENCIL STAR. > > However, in some parts of my code, I need to update ghost points at east > east, west west etc. > > DMGlobalToLocalBegin/end can only update east, west etc. What's the > easiest way to update these additional points? Do I have to use MPI_Send? > > Also, how is the memory allocated? I think the east east values will be > outside the local vector, is that so? > > Thank you. > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jun 15 20:31:57 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 15 Jun 2012 21:31:57 -0400 Subject: [petsc-users] Obtaining more ghost points at EE, SS etc using DM In-Reply-To: References: <4FDBDA29.7020809@gmail.com> Message-ID: <4FDBE20D.3090106@gmail.com> On 15/6/2012 9:04 PM, Jed Brown wrote: > If you need corner points, use DMDA_STENCIL_BOX. Sorry? I don't mean corner points, but points to the east of the east cell, or the west of the west cell ie. instead of only 1 ghost cell to the east of the boundary, I need 1 more which is east of the east cell. > > On Fri, Jun 15, 2012 at 7:58 PM, TAY wee-beng > wrote: > > Hi, > > I'm trying to implement DM in my code. So far, I found that it is > very convenient to use its features. I am currently using DMDA > STENCIL STAR. > > However, in some parts of my code, I need to update ghost points > at east east, west west etc. > > DMGlobalToLocalBegin/end can only update east, west etc. What's > the easiest way to update these additional points? Do I have to > use MPI_Send? > > Also, how is the memory allocated? I think the east east values > will be outside the local vector, is that so? > > Thank you. > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 15 20:32:55 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 15 Jun 2012 20:32:55 -0500 Subject: [petsc-users] Obtaining more ghost points at EE, SS etc using DM In-Reply-To: <4FDBE20D.3090106@gmail.com> References: <4FDBDA29.7020809@gmail.com> <4FDBE20D.3090106@gmail.com> Message-ID: On Fri, Jun 15, 2012 at 8:31 PM, TAY wee-beng wrote: > > On 15/6/2012 9:04 PM, Jed Brown wrote: > > If you need corner points, use DMDA_STENCIL_BOX. > > > Sorry? I don't mean corner points, but points to the east of the east > cell, or the west of the west cell ie. instead of only 1 ghost cell to the > east of the boundary, I need 1 more which is east of the east cell. > Then use a stencil width of 2. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Fri Jun 15 20:48:53 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 15 Jun 2012 21:48:53 -0400 Subject: [petsc-users] Obtaining more ghost points at EE, SS etc using DM In-Reply-To: References: <4FDBDA29.7020809@gmail.com> <4FDBE20D.3090106@gmail.com> Message-ID: <4FDBE605.9020405@gmail.com> On 15/6/2012 9:32 PM, Jed Brown wrote: > On Fri, Jun 15, 2012 at 8:31 PM, TAY wee-beng > wrote: > > > On 15/6/2012 9:04 PM, Jed Brown wrote: >> If you need corner points, use DMDA_STENCIL_BOX. > > Sorry? I don't mean corner points, but points to the east of the > east cell, or the west of the west cell ie. instead of only 1 > ghost cell to the east of the boundary, I need 1 more which is > east of the east cell. > > > Then use a stencil width of 2. Ok I see the option now. thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Jun 16 08:11:30 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 16 Jun 2012 09:11:30 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid Message-ID: <4FDC8602.5060304@gmail.com> Hi, I'm abit unclear about dof used in DMDACreate3d. If I have u,v,w for my CFD code, does it mean that I should have dof = 3? Or can i create 3 local/global vector using DMCreateGlobalVector and DMCreateLocalVector to define u,v,w? Does it matter if I'm using staggered grids? Also, if I am only using PETSc for u,v,w (and not pressure, which uses HYPRE), does it mean a difference? I remember there 's an example using staggered grid somewhere but I can't find it now. May I know which PETSc 's example it is? Thanks! -- Yours sincerely, TAY wee-beng From jedbrown at mcs.anl.gov Sat Jun 16 08:24:53 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 16 Jun 2012 08:24:53 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDC8602.5060304@gmail.com> References: <4FDC8602.5060304@gmail.com> Message-ID: It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). On Jun 16, 2012 8:11 AM, "TAY wee-beng" wrote: > Hi, > > I'm abit unclear about dof used in DMDACreate3d. If I have u,v,w for my > CFD code, does it mean that I should have dof = 3? Or can i create 3 > local/global vector using DMCreateGlobalVector and DMCreateLocalVector to > define u,v,w? > > Does it matter if I'm using staggered grids? Also, if I am only using > PETSc for u,v,w (and not pressure, which uses HYPRE), does it mean a > difference? > > I remember there 's an example using staggered grid somewhere but I can't > find it now. May I know which PETSc 's example it is? > > Thanks! > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Jun 16 08:37:36 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 16 Jun 2012 08:37:36 -0500 Subject: [petsc-users] How to implement projection preconditioner? In-Reply-To: <1339843486.59752.YahooMailNeo@web28703.mail.ir2.yahoo.com> References: <1339174723.57548.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339602854.42453.YahooMailNeo@web28705.mail.ir2.yahoo.com> <1339788229.6453.YahooMailNeo@web28701.mail.ir2.yahoo.com> <1339790858.47311.YahooMailNeo@web28703.mail.ir2.yahoo.com> <1339843486.59752.YahooMailNeo@web28703.mail.ir2.yahoo.com> Message-ID: \bar C is up to you to define, so that's just notation. You can tell PCMG to use your own residual if you want to do some crazy residual. Otherwise just use the matrix and preconditioning matrix to get the algorithm you want. Always look at -ksp_view to see what is going on. On Jun 16, 2012 5:44 AM, "Abdul Hanan Sheikh" wrote: > Dear , > > Thanks for simplifiying. I got it well :D > > I am still stuck there to implement Prec1 = \bar C b + (b - \bar A \bar C > b). > My interest is to get this "Prec" implemented in Petsc. > I see this as a projection preconditioner. > > Starting with; PCMG on system Ax = b with no presmoothing, and Richardson > as postmoothing serves > as Prec = C + (I - AC) , (right ? ) . Now what should I do to get Prec1 = \bar > C + (I - \bar A \bar C ). > > thansk, Abdul > > ------------------------------ > *From:* Jed Brown > *To:* Abdul Hanan Sheikh > *Cc:* PETSc users list > *Sent:* Friday, 15 June 2012, 23:58 > *Subject:* Re: [petsc-users] How to implement projection preconditioner? > > Let's get even simpler, suppose I start with > > A x = b > > The left-preconditioned form of this system is *not* > > M^{-1} A x = b > > it is > > M^{-1} A x = M^{-1} b > > > Run that through your analysis and you'll find that what your asking for > doesn't make sense. It often helps keep things straight if you consider > that the residual and solution do not lie in the same space, in which case > "no preconditioning" is the "trivial" map from the residual space back to > the state space. > > On Fri, Jun 15, 2012 at 3:07 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > Thank you for reply! > I didnt get , what does it mean by applying the preconditioner to the > right hand side, making it look just like a smoother ? > > More simply, how can I implement Prec = \bar C b + (b - \bar A \bar C b) > where \bar C = P (M_H^-1 A_H)^-1 R > and \bar A = M^-1 A. > Note: we know how to implement Prec = C b + (b - A C b). > > Many thanks, Abdul > > ------------------------------ > *From:* Jed Brown > *To:* Abdul Hanan Sheikh > *Cc:* PETSc users list > *Sent:* Friday, 15 June 2012, 21:36 > > *Subject:* Re: [petsc-users] How to implement projection preconditioner? > > On Fri, Jun 15, 2012 at 2:23 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > I recall my previous thread; copy and paste here, and then I try to ask: > > Suppose C = P*A_H^{-1}*R is the coarse grid solver. Then with a > post-smoother S, application of the preconditioner to the vector "b" would > be C b + S (b - A C b). If we drop the post-smoother by setting S = I, We > get Prec = C b + b - A C b. Note that there is no presmoother. > > This is the preconditioner I have used before, in PCMG framework, with no > pre-smoothing and Richardson(without any preconditioner) as post smoother. > > I need to apply the updated version of above Prec = C b + (b - A C b), > where A is replaced by M^-1 A , and A_H by M_H^-1 A_H . > > > With left preconditioning, you also have to apply the preconditioner to > the right hand side, thus making it look just like a smoother. > > > > Now does it make sense : > > 1) incorporating M as only preconditioner in presmoother will replace A > by M^-1 A in Prec = C b + (b - A C b) ? > > > No, it uses the preconditioner the way that preconditioners work, see > above. > > > 2) incorporating M as preconditioner to CG-solver will replace A_H by > (M_H^-1 A_H ) ? I am kinda of sure about this point 2. > > > C already contains A_H^{-1}. Replacing that by M_H^{-1} is an approximate > coarse level solve (that's okay and seems to be what you want, just not > what you have written, which is not a consistent method). > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Sun Jun 17 05:27:53 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Sun, 17 Jun 2012 18:27:53 +0800 Subject: [petsc-users] How to implement multigrid preconditioning without DMMG Message-ID: Hi everyone, I has been developing some solvers with multigrid preconditioning built on the top of Object DMMG. Now, I update Petsc from 3.2 to 3.3, and find that there is no DMMG. I try to modify my code so that it can run on new Petsc. But I don't know how to transfer my code. If directly modify code, it will take me a lot of time. Even further, I cann't find any example on how to use "PCMG". Regards, ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Jun 17 07:26:52 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 17 Jun 2012 08:26:52 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: References: <4FDC8602.5060304@gmail.com> Message-ID: <4FDDCD0C.5080302@gmail.com> On 16/6/2012 9:24 AM, Jed Brown wrote: > > It depends how you want to solve the problem. I usually group all dofs > together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or > 31?). > I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. Supposed I have a field - u,v,w,p, so in order to use them, I do the following: / type field real u,v,w,p end type field type(field), pointer :: field1(:,:,:) -> make a derived variable/ Also: / Vec field_local,field_global call DMDACreate3d with dof = 4 call DMCreateGlobalVector(da, field_local,ierr) call DMCreateLocalVector(da,field_global,ierr) call DMGetLocalVector(da, field_local,ierr) -> To insert values call DMDAVecGetArrayF90(da, field_local,field1,ierr) do k = zs, zs + zm - 1 do j = ys, ys + ym -1 do i = xs, xs + xm - 1 field1(i,j,k)%u = ... -> evaluate u,v,w,p etc end do end do call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) call DMRestoreLocalVector(da,field_local,ierr)/ Is this the correct way? Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) For b, do I duplicate a Vec similar to field_local? What about matrix A? Do I use the MatSetValuesStencil? Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? Thanks! > On Jun 16, 2012 8:11 AM, "TAY wee-beng" > wrote: > > Hi, > > I'm abit unclear about dof used in DMDACreate3d. If I have u,v,w > for my CFD code, does it mean that I should have dof = 3? Or can i > create 3 local/global vector using DMCreateGlobalVector and > DMCreateLocalVector to define u,v,w? > > Does it matter if I'm using staggered grids? Also, if I am only > using PETSc for u,v,w (and not pressure, which uses HYPRE), does > it mean a difference? > > I remember there 's an example using staggered grid somewhere but > I can't find it now. May I know which PETSc 's example it is? > > Thanks! > > -- > Yours sincerely, > > TAY wee-beng > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Jun 17 11:57:27 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 17 Jun 2012 11:57:27 -0500 Subject: [petsc-users] How to implement multigrid preconditioning without DMMG In-Reply-To: References: Message-ID: <16DE16A9-7FFE-4883-9E02-94FC8270109F@mcs.anl.gov> Look at the examples, for nonlinear problems look at src/snes/examples/tutorials/ex19.c for linear problems look at src/ksp/ksp/examples/tutorials/ ex45.c they are both translations of what use to be DMMG examples so should show you exactly what needs to be done. Barry On Jun 17, 2012, at 5:27 AM, fdkong wrote: > Hi everyone, > > I has been developing some solvers with multigrid preconditioning built on the top of Object DMMG. Now, I update Petsc from 3.2 to 3.3, and find that there is no DMMG. I try to modify my code so that it can run on new Petsc. But I don't know how to transfer my code. If directly modify code, it will take me a lot of time. Even further, I cann't find any example on how to use "PCMG". > > Regards, > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > From jedbrown at mcs.anl.gov Sun Jun 17 13:33:30 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 17 Jun 2012 13:33:30 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDDCD0C.5080302@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> Message-ID: On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: > > On 16/6/2012 9:24 AM, Jed Brown wrote: > > It depends how you want to solve the problem. I usually group all dofs > together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). > > > I tried to understand ex30. I have some questions since it's in C and I'm > using to Fortran programming. > This looks about right, see src/dm/examples/tutorials/ex11f90.F. > > Supposed I have a field - u,v,w,p, so in order to use them, I do the > following: > * > type field > > real u,v,w,p > > end type field > > type(field), pointer :: field1(:,:,:) -> make a derived variable* > > Also: > * > Vec field_local,field_global > > call DMDACreate3d with dof = 4 > > call DMCreateGlobalVector(da, field_local,ierr) > > call DMCreateLocalVector(da,field_global,ierr) > > call DMGetLocalVector(da, field_local,ierr) -> To insert values > > call DMDAVecGetArrayF90(da, field_local,field1,ierr) > > do k = zs, zs + zm - 1 > > do j = ys, ys + ym -1 > > do i = xs, xs + xm - 1 > > field1(i,j,k)%u = ... -> evaluate u,v,w,p etc > > end do > > end do > > call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) > > call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) > > call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) > > call DMRestoreLocalVector(da,field_local,ierr)* > > Is this the correct way? > > Also, supposed I now want to solve my u,v,w momentum eqns. Although > they're not coupled together, I believe it's faster if I assemble them into > 1 big matrix. > > So for Ax = b, x = > (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) > > For b, do I duplicate a Vec similar to field_local? > > What about matrix A? Do I use the MatSetValuesStencil? > Yes > > Lastly, the type field contains u,v,w and p. However, I'm only solving > u,v,w. Do I have to skip some values or use identity matrix to solve it? > Why not make a field that contains only u,v,w. I don't see what you're trying to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Mon Jun 18 06:58:24 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 18 Jun 2012 13:58:24 +0200 Subject: [petsc-users] How to specify a preconditioner matrix for pc_fieldsplit_type Schur? Message-ID: <4FDF17E0.9030903@tu-dresden.de> Can some of you give me an example how to specify a preconditioner matrix in the case I have a fieldsplit preconditioner of Schur type? Or in the notation of the manual, how to set Sp in ksp(S, Sp) [see manual p. 87]. Thomas From fd.kong at siat.ac.cn Mon Jun 18 08:11:30 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Mon, 18 Jun 2012 21:11:30 +0800 Subject: [petsc-users] How to implement multigrid preconditioning without DMMG Message-ID: Hi Barry, Thank you very much! Now, mg preconditioning works. Date: Sun, 17 Jun 2012 11:57:27 -0500 > Look at the examples, for nonlinear problems look at src/snes/examples/tutorials/ex19.c for linear problems look at src/ksp/ksp/examples/tutorials/ ex45.c they are both translations of what use to be DMMG examples so should show you exactly what needs to be done. > Barry >On Jun 17, 2012, at 5:27 AM, fdkong wrote: >> Hi everyone, >> >> I has been developing some solvers with multigrid preconditioning built on the top of Object DMMG. Now, I update Petsc from 3.2 to 3.3, and find that there is no DMMG. I try to modify my code so that it can run on new Petsc. But I don't know how to transfer my code. If directly modify code, it will take me a lot of time. Even further, I cann't find any example on how to use "PCMG". >> >> Regards, >> ------------------ >> Fande Kong >> ShenZhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 18 12:36:21 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 18 Jun 2012 09:36:21 -0800 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type In-Reply-To: References: Message-ID: On Mon, Mar 19, 2012 at 5:29 AM, Klaij, Christiaan wrote: > > It looks like PCFieldSplitSetSchurFactorizationType() is missing. > > Would be nice to have. I don't know if I ever pointed this out to you. It's in 3.3. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/6d8eec http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ead058 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 18 12:39:45 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 18 Jun 2012 09:39:45 -0800 Subject: [petsc-users] How to specify a preconditioner matrix for pc_fieldsplit_type Schur? In-Reply-To: <4FDF17E0.9030903@tu-dresden.de> References: <4FDF17E0.9030903@tu-dresden.de> Message-ID: On Mon, Jun 18, 2012 at 3:58 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Can some of you give me an example how to specify a preconditioner matrix > in the case I have a fieldsplit preconditioner of Schur type? Or in the > notation of the manual, how to set Sp in ksp(S, Sp) [see manual p. 87]. This is the most straight forward way. There are other ways if this seems especially unnatural for you. http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCFieldSplitSchurPrecondition.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaetano at email.virginia.edu Mon Jun 18 16:54:55 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Mon, 18 Jun 2012 17:54:55 -0400 Subject: [petsc-users] Timestep Splitting Methods Message-ID: I was wondering whether there is a suggested strategy to implement something like a timestep splitting method for systems of PDE's (NS) in Petsc. By timestep splitting I mean this: advance independently in time each term of the equations, then add all the solutions to get the total effects. >From my basic understanding of the TS object, I could basically define only one RHS function. Is creating several TS objects a recommended approach? What are the downsides of this? Many Thanks, --g From yifli82 at gmail.com Mon Jun 18 17:21:27 2012 From: yifli82 at gmail.com (Yifei Li) Date: Mon, 18 Jun 2012 18:21:27 -0400 Subject: [petsc-users] make test failed Message-ID: Hi all, I got the following error when running 'make test': Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and PETSC_ARCH=arch-mswin-c-debug Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near unexpected token 'elif' 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" ]; then Can someone help me with this? Thanks Yifei -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Jun 18 17:26:34 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 18 Jun 2012 14:26:34 -0800 Subject: [petsc-users] Timestep Splitting Methods In-Reply-To: References: Message-ID: We have two "splitting" methods in PETSc, additive Runge-Kutta IMEX (a nonlinearly implicit method) and Rosenbrock-W (a more convenient linearly implicit method). Unlike almost all ad-hoc splitting methods, these have high order accuracy and provably good stability properties. Read the man pages and use the IFunction/RHSFunction/IJacobian interface, see src/ts/examples/tutorials/ex22.c and other examples. On Mon, Jun 18, 2012 at 1:54 PM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > I was wondering whether there is a suggested strategy to implement > something like a timestep splitting method for systems of PDE's (NS) > in Petsc. > By timestep splitting I mean this: advance independently in time each > term of the equations, then add all the solutions to get the total > effects. > From my basic understanding of the TS object, I could basically define > only one RHS function. > Is creating several TS objects a recommended approach? What are the > downsides of this? > > Many Thanks, > > --g > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fd.kong at siat.ac.cn Mon Jun 18 20:04:17 2012 From: fd.kong at siat.ac.cn (=?gb18030?B?ZmRrb25n?=) Date: Tue, 19 Jun 2012 09:04:17 +0800 Subject: [petsc-users] Can c++ SieveMesh support 64-bit? Message-ID: Hi Mat, I have been developing some codes based-on c++ sievemesh (your old version code). I want to know the following questions: (1) You have added a new c sievemesh. Will the old c++ version be remove in future? (2) Have you ever made some tests on 64-bit computer? Whether c++ sieveMesh can run on 64-bit computer? I want to use 64-bit integer. Regards, ------------------ Fande Kong ShenZhen Institutes of Advanced Technology Chinese Academy of Sciences -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jun 18 20:30:23 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 18 Jun 2012 21:30:23 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> Message-ID: <4FDFD62F.3030702@gmail.com> On 17/6/2012 2:33 PM, Jed Brown wrote: > On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng > wrote: > > > On 16/6/2012 9:24 AM, Jed Brown wrote: >> >> It depends how you want to solve the problem. I usually group all >> dofs together. There is a 2D Stokes+Thermodynamics example in >> SNES ex30 (or 31?). >> > > I tried to understand ex30. I have some questions since it's in C > and I'm using to Fortran programming. > > > This looks about right, see src/dm/examples/tutorials/ex11f90.F. I tried to build and run ex11f90 in linux. However, it's not working: I have attached the run and valgrind output: run output: / [wtay at hpc12:tutorials]$ ./ex11f90 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. --------------------------------------------------------------------------/ valgrind output: / ==6027== Memcheck, a memory error detector ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==6027== Command: ex11f90 ==6027== ==6027== Invalid read of size 8 ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) ==6027== by 0x95C39FE: opal_init (opal_init.c:307) ==6027== by 0x95815A4: orte_init (orte_init.c:78) ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) ==6027== by 0x95C39FE: opal_init (opal_init.c:307) ==6027== by 0x95815A4: orte_init (orte_init.c:78) ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) ==6027== by 0x955938F: PMPI_Init (pinit.c:84) ==6027== ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) ==6027== by 0x955938F: PMPI_Init (pinit.c:84) ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) ==6027== by 0x955938F: PMPI_Init (pinit.c:84) ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) ==6027== ==6027== Invalid read of size 8 ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) ==6027== by 0xEE1CC2F: ??? ==6027== by 0xEE5A0AF: ??? ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) ==6027== by 0x7FEFFFC13: ??? ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd ==6027== / > > Supposed I have a field - u,v,w,p, so in order to use them, I do > the following: > / > type field > > real u,v,w,p > > end type field > > type(field), pointer :: field1(:,:,:) -> make a derived variable/ > > Also: > / > Vec field_local,field_global > > call DMDACreate3d with dof = 4 > > call DMCreateGlobalVector(da, field_local,ierr) > > call DMCreateLocalVector(da,field_global,ierr) > > call DMGetLocalVector(da, field_local,ierr) -> To insert values > > call DMDAVecGetArrayF90(da, field_local,field1,ierr) > > do k = zs, zs + zm - 1 > > do j = ys, ys + ym -1 > > do i = xs, xs + xm - 1 > > field1(i,j,k)%u = ... -> evaluate u,v,w,p etc > > end do > > end do > > call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) > > call > DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) > > call > DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) > > call DMRestoreLocalVector(da,field_local,ierr)/ > > Is this the correct way? > > Also, supposed I now want to solve my u,v,w momentum eqns. > Although they're not coupled together, I believe it's faster if I > assemble them into 1 big matrix. > > So for Ax = b, x = > (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) > > For b, do I duplicate a Vec similar to field_local? > > What about matrix A? Do I use the MatSetValuesStencil? > > > Yes > > > Lastly, the type field contains u,v,w and p. However, I'm only > solving u,v,w. Do I have to skip some values or use identity > matrix to solve it? > > > Why not make a field that contains only u,v,w. I don't see what you're > trying to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 18 20:52:44 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 18 Jun 2012 19:52:44 -0600 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDFD62F.3030702@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> Message-ID: On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: > > On 17/6/2012 2:33 PM, Jed Brown wrote: > > On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: > >> >> On 16/6/2012 9:24 AM, Jed Brown wrote: >> >> It depends how you want to solve the problem. I usually group all dofs >> together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >> >> >> I tried to understand ex30. I have some questions since it's in C and >> I'm using to Fortran programming. >> > > This looks about right, see src/dm/examples/tutorials/ex11f90.F. > > > I tried to build and run ex11f90 in linux. However, it's not working: > > I have attached the run and valgrind output: > It appears to be a problem with your MPI. Matt > run output: > * > [wtay at hpc12:tutorials]$ ./ex11f90 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find > memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 > -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 > 03:22:52 2012 > [0]PETSC ERROR: Libraries linked from > /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib > [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 > [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ > --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ > --with-debugging=1 --download-hypre=1 > --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 > --with-shared-libraries > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown directory > unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > * > > valgrind output: > * > ==6027== Memcheck, a memory error detector > ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. > ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info > ==6027== Command: ex11f90 > ==6027== > ==6027== Invalid read of size 8 > ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/ > libc-2.12.so) > ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) > ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) > ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) > ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) > ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) > ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) > ==6027== by 0x95DB999: mca_base_component_find > (mca_base_component_find.c:301) > ==6027== by 0x95DC4B0: mca_base_components_open > (mca_base_components_open.c:128) > ==6027== by 0x95F7CE7: opal_paffinity_base_open > (paffinity_base_open.c:112) > ==6027== by 0x95C39FE: opal_init (opal_init.c:307) > ==6027== by 0x95815A4: orte_init (orte_init.c:78) > ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd > ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) > ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) > ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) > ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) > ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) > ==6027== by 0x95DB999: mca_base_component_find > (mca_base_component_find.c:301) > ==6027== by 0x95DC4B0: mca_base_components_open > (mca_base_components_open.c:128) > ==6027== by 0x95F7CE7: opal_paffinity_base_open > (paffinity_base_open.c:112) > ==6027== by 0x95C39FE: opal_init (opal_init.c:307) > ==6027== by 0x95815A4: orte_init (orte_init.c:78) > ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== > ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) > ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) > ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) > ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) > ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) > ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) > ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) > ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) > ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) > ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) > ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) > ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd > ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) > ==6027== by 0x95C4F22: opal_dss_buffer_extend > (dss_internal_functions.c:63) > ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) > ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries > (grpcomm_base_modex.c:861) > ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) > ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) > ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) > ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) > ==6027== by 0x4087AB: main (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== > ==6027== Invalid read of size 8 > ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) > ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) > ==6027== by 0xEE1CC2F: ??? > ==6027== by 0xEE5A0AF: ??? > ==6027== by 0x6F5C9F: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x4962FF: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x6F5C9F: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x7FEFFFC13: ??? > ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or > (recently) free'd > ==6027== * > > > > > > >> >> Supposed I have a field - u,v,w,p, so in order to use them, I do the >> following: >> * >> type field >> >> real u,v,w,p >> >> end type field >> >> type(field), pointer :: field1(:,:,:) -> make a derived variable* >> >> Also: >> * >> Vec field_local,field_global >> >> call DMDACreate3d with dof = 4 >> >> call DMCreateGlobalVector(da, field_local,ierr) >> >> call DMCreateLocalVector(da,field_global,ierr) >> >> call DMGetLocalVector(da, field_local,ierr) -> To insert values >> >> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >> >> do k = zs, zs + zm - 1 >> >> do j = ys, ys + ym -1 >> >> do i = xs, xs + xm - 1 >> >> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >> >> end do >> >> end do >> >> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >> >> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >> >> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >> >> call DMRestoreLocalVector(da,field_local,ierr)* >> >> Is this the correct way? >> >> Also, supposed I now want to solve my u,v,w momentum eqns. Although >> they're not coupled together, I believe it's faster if I assemble them into >> 1 big matrix. >> >> So for Ax = b, x = >> (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >> >> For b, do I duplicate a Vec similar to field_local? >> >> What about matrix A? Do I use the MatSetValuesStencil? >> > > Yes > > >> >> Lastly, the type field contains u,v,w and p. However, I'm only solving >> u,v,w. Do I have to skip some values or use identity matrix to solve it? >> > > Why not make a field that contains only u,v,w. I don't see what you're > trying to do. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jun 18 20:54:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 18 Jun 2012 19:54:23 -0600 Subject: [petsc-users] Can c++ SieveMesh support 64-bit? In-Reply-To: References: Message-ID: On Mon, Jun 18, 2012 at 7:04 PM, fdkong wrote: > Hi Mat, > > I have been developing some codes based-on c++ sievemesh (your old version > code). I want to know the following questions: > > (1) You have added a new c sievemesh. Will the old c++ version be remove > in future? > Yes, but not soon. However, I recommend making a plan for switching, since you get so many benefits including much better solver integration. Take a look at SNES ex62. > (2) Have you ever made some tests on 64-bit computer? Whether c++ > sieveMesh can run on 64-bit computer? I want to use 64-bit integer. > It will work on a 64-bit computer. Both version use PetscInt. Matt > Regards, > ** > ------------------ > Fande Kong > ShenZhen Institutes of Advanced Technology > Chinese Academy of Sciences > ** > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jun 18 21:21:15 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 18 Jun 2012 22:21:15 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> Message-ID: <4FDFE21B.30903@gmail.com> On 18/6/2012 9:52 PM, Matthew Knepley wrote: > On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng > wrote: > > > On 17/6/2012 2:33 PM, Jed Brown wrote: >> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng > > wrote: >> >> >> On 16/6/2012 9:24 AM, Jed Brown wrote: >>> >>> It depends how you want to solve the problem. I usually >>> group all dofs together. There is a 2D Stokes+Thermodynamics >>> example in SNES ex30 (or 31?). >>> >> >> I tried to understand ex30. I have some questions since it's >> in C and I'm using to Fortran programming. >> >> >> This looks about right, see src/dm/examples/tutorials/ex11f90.F. > > I tried to build and run ex11f90 in linux. However, it's not working: > > I have attached the run and valgrind output: > > > It appears to be a problem with your MPI. > > Matt Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. > > run output: > / > [wtay at hpc12:tutorials]$ ./ex11f90 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > [0]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC > ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X > to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Signal received! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 > 17:50:32 2012 -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue > Jun 19 03:22:52 2012 > [0]PETSC ERROR: Libraries linked from > /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib > [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/opt/openmpi-1.5.3/ > --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ > --with-debugging=1 --download-hypre=1 > --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug > --known-mpi-shared=1 --with-shared-libraries > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: User provided function() line 0 in unknown > directory unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > --------------------------------------------------------------------------/ > > valgrind output: > / > ==6027== Memcheck, a memory error detector > ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward > et al. > ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for > copyright info > ==6027== Command: ex11f90 > ==6027== > ==6027== Invalid read of size 8 > ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in > /lib64/libc-2.12.so ) > ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so > ) > ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so > ) > ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) > ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) > ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) > ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) > ==6027== by 0x95DB999: mca_base_component_find > (mca_base_component_find.c:301) > ==6027== by 0x95DC4B0: mca_base_components_open > (mca_base_components_open.c:128) > ==6027== by 0x95F7CE7: opal_paffinity_base_open > (paffinity_base_open.c:112) > ==6027== by 0x95C39FE: opal_init (opal_init.c:307) > ==6027== by 0x95815A4: orte_init (orte_init.c:78) > ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 > alloc'd > ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) > ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) > ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) > ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) > ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) > ==6027== by 0x95DB999: mca_base_component_find > (mca_base_component_find.c:301) > ==6027== by 0x95DC4B0: mca_base_components_open > (mca_base_components_open.c:128) > ==6027== by 0x95F7CE7: opal_paffinity_base_open > (paffinity_base_open.c:112) > ==6027== by 0x95C39FE: opal_init (opal_init.c:307) > ==6027== by 0x95815A4: orte_init (orte_init.c:78) > ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== > ==6027== Syscall param writev(vector[...]) points to uninitialised > byte(s) > ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so > ) > ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler > (oob_tcp_msg.c:249) > ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) > ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) > ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) > ==6027== by 0xC3955FF: orte_rml_oob_send_buffer > (rml_oob_send.c:270) > ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) > ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) > ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) > ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) > ==6027== Address 0xed30cd1 is 161 bytes inside a block of size > 256 alloc'd > ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) > ==6027== by 0x95C4F22: opal_dss_buffer_extend > (dss_internal_functions.c:63) > ==6027== by 0x95C5A64: opal_dss_copy_payload > (dss_load_unload.c:164) > ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries > (grpcomm_base_modex.c:861) > ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) > ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) > ==6027== by 0x955938F: PMPI_Init (pinit.c:84) > ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) > ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) > ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) > ==6027== by 0x4087AB: main (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== > ==6027== Invalid read of size 8 > ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) > ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) > ==6027== by 0xEE1CC2F: ??? > ==6027== by 0xEE5A0AF: ??? > ==6027== by 0x6F5C9F: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x4962FF: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x6F5C9F: ??? (in > /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) > ==6027== by 0x7FEFFFC13: ??? > ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or > (recently) free'd > ==6027== / > > > >> >> Supposed I have a field - u,v,w,p, so in order to use them, I >> do the following: >> / >> type field >> >> real u,v,w,p >> >> end type field >> >> type(field), pointer :: field1(:,:,:) -> make a derived >> variable/ >> >> Also: >> / >> Vec field_local,field_global >> >> call DMDACreate3d with dof = 4 >> >> call DMCreateGlobalVector(da, field_local,ierr) >> >> call DMCreateLocalVector(da,field_global,ierr) >> >> call DMGetLocalVector(da, field_local,ierr) -> To >> insert values >> >> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >> >> do k = zs, zs + zm - 1 >> >> do j = ys, ys + ym -1 >> >> do i = xs, xs + xm - 1 >> >> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >> >> end do >> >> end do >> >> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >> >> call >> DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >> >> call >> DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >> >> call DMRestoreLocalVector(da,field_local,ierr)/ >> >> Is this the correct way? >> >> Also, supposed I now want to solve my u,v,w momentum eqns. >> Although they're not coupled together, I believe it's faster >> if I assemble them into 1 big matrix. >> >> So for Ax = b, x = >> (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... >> ) >> >> For b, do I duplicate a Vec similar to field_local? >> >> What about matrix A? Do I use the MatSetValuesStencil? >> >> >> Yes >> >> >> Lastly, the type field contains u,v,w and p. However, I'm >> only solving u,v,w. Do I have to skip some values or use >> identity matrix to solve it? >> >> >> Why not make a field that contains only u,v,w. I don't see what >> you're trying to do. > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 18 21:36:57 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Jun 2012 21:36:57 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDFE21B.30903@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> Message-ID: <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f Barry On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: > > On 18/6/2012 9:52 PM, Matthew Knepley wrote: >> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >> >> On 17/6/2012 2:33 PM, Jed Brown wrote: >>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>> >>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>> >>> >>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>> >>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >> >> I tried to build and run ex11f90 in linux. However, it's not working: >> >> I have attached the run and valgrind output: >> >> It appears to be a problem with your MPI. >> >> Matt > > Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >> >> run output: >> >> [wtay at hpc12:tutorials]$ ./ex11f90 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >> [0]PETSC ERROR: likely location of problem given in stack below >> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [0]PETSC ERROR: INSTEAD the line number of the start of the function >> [0]PETSC ERROR: is given. >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Signal received! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >> -------------------------------------------------------------------------- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 59. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -------------------------------------------------------------------------- >> >> valgrind output: >> >> ==6027== Memcheck, a memory error detector >> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >> ==6027== Command: ex11f90 >> ==6027== >> ==6027== Invalid read of size 8 >> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >> ==6027== >> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >> ==6027== >> ==6027== Invalid read of size 8 >> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >> ==6027== by 0xEE1CC2F: ??? >> ==6027== by 0xEE5A0AF: ??? >> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >> ==6027== by 0x7FEFFFC13: ??? >> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >> ==6027== >> >> >> >>> >>> >>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>> >>> type field >>> >>> real u,v,w,p >>> >>> end type field >>> >>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>> >>> Also: >>> >>> Vec field_local,field_global >>> >>> call DMDACreate3d with dof = 4 >>> >>> call DMCreateGlobalVector(da, field_local,ierr) >>> >>> call DMCreateLocalVector(da,field_global,ierr) >>> >>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>> >>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>> >>> do k = zs, zs + zm - 1 >>> >>> do j = ys, ys + ym -1 >>> >>> do i = xs, xs + xm - 1 >>> >>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>> >>> end do >>> >>> end do >>> >>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>> >>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>> >>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>> >>> call DMRestoreLocalVector(da,field_local,ierr) >>> >>> Is this the correct way? >>> >>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>> >>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>> >>> For b, do I duplicate a Vec similar to field_local? >>> >>> What about matrix A? Do I use the MatSetValuesStencil? >>> >>> Yes >>> >>> >>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>> >>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener From zonexo at gmail.com Mon Jun 18 21:54:03 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 18 Jun 2012 22:54:03 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> Message-ID: <4FDFE9CB.1040303@gmail.com> On 18/6/2012 10:36 PM, Barry Smith wrote: > Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f > > Barry ex19 and ex5f both run w/o problem. [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex19 lid velocity = 0.0625, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex19 lid velocity = 0.0625, prandtl # = 1, grashof # = 1 Number of SNES iterations = 2 [wtay at hpc12:tutorials]$ ./ex5f Number of SNES iterations = 4 [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex5f Number of SNES iterations = 4 [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex5f Number of SNES iterations = 4 [wtay at hpc12:tutorials]$ The problem seems to lie in DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90. I have reported this problem before when I convert ex29 from C to Fortran. It works in Vs2008 windows but not in linux. I wonder if it has some bugs or compatibility problem. > > On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: > >> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>> >>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>> >>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>> >>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>> >>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>> I tried to build and run ex11f90 in linux. However, it's not working: >>> >>> I have attached the run and valgrind output: >>> >>> It appears to be a problem with your MPI. >>> >>> Matt >> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>> >>> run output: >>> >>> [wtay at hpc12:tutorials]$ ./ex11f90 >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Signal received! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>> -------------------------------------------------------------------------- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -------------------------------------------------------------------------- >>> >>> valgrind output: >>> >>> ==6027== Memcheck, a memory error detector >>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>> ==6027== Command: ex11f90 >>> ==6027== >>> ==6027== Invalid read of size 8 >>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>> ==6027== >>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>> ==6027== >>> ==6027== Invalid read of size 8 >>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>> ==6027== by 0xEE1CC2F: ??? >>> ==6027== by 0xEE5A0AF: ??? >>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>> ==6027== by 0x7FEFFFC13: ??? >>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>> ==6027== >>> >>> >>> >>>> >>>> >>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>> >>>> type field >>>> >>>> real u,v,w,p >>>> >>>> end type field >>>> >>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>> >>>> Also: >>>> >>>> Vec field_local,field_global >>>> >>>> call DMDACreate3d with dof = 4 >>>> >>>> call DMCreateGlobalVector(da, field_local,ierr) >>>> >>>> call DMCreateLocalVector(da,field_global,ierr) >>>> >>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>> >>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>> >>>> do k = zs, zs + zm - 1 >>>> >>>> do j = ys, ys + ym -1 >>>> >>>> do i = xs, xs + xm - 1 >>>> >>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>> >>>> end do >>>> >>>> end do >>>> >>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>> >>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>> >>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>> >>>> call DMRestoreLocalVector(da,field_local,ierr) >>>> >>>> Is this the correct way? >>>> >>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>> >>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>> >>>> For b, do I duplicate a Vec similar to field_local? >>>> >>>> What about matrix A? Do I use the MatSetValuesStencil? >>>> >>>> Yes >>>> >>>> >>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>> >>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener From bsmith at mcs.anl.gov Mon Jun 18 21:58:30 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Jun 2012 21:58:30 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDFE9CB.1040303@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> <4FDFE9CB.1040303@gmail.com> Message-ID: On Jun 18, 2012, at 9:54 PM, TAY wee-beng wrote: > > On 18/6/2012 10:36 PM, Barry Smith wrote: >> Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f >> >> Barry > ex19 and ex5f both run w/o problem. > > [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex19 > lid velocity = 0.0625, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex19 > lid velocity = 0.0625, prandtl # = 1, grashof # = 1 > Number of SNES iterations = 2 > > [wtay at hpc12:tutorials]$ ./ex5f > Number of SNES iterations = 4 > [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex5f > Number of SNES iterations = 4 > [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex5f > Number of SNES iterations = 4 > [wtay at hpc12:tutorials]$ > > The problem seems to lie in DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90. I have reported this problem before when I convert ex29 from C to Fortran. It works in Vs2008 windows but not in linux. I wonder if it has some bugs or compatibility problem. Then likely it is an Intel compiler bug. Does this happen with the gnu gfortran compiler combined with gcc as the c compiler? Barry > >> >> On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: >> >>> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>>> >>>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>>> >>>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>>> >>>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>>> >>>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>>> I tried to build and run ex11f90 in linux. However, it's not working: >>>> >>>> I have attached the run and valgrind output: >>>> >>>> It appears to be a problem with your MPI. >>>> >>>> Matt >>> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>>> >>>> run output: >>>> >>>> [wtay at hpc12:tutorials]$ ./ex11f90 >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [0]PETSC ERROR: Signal received! >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>> -------------------------------------------------------------------------- >>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>> with errorcode 59. >>>> >>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> You may or may not see output from other processes, depending on >>>> exactly when Open MPI kills them. >>>> -------------------------------------------------------------------------- >>>> >>>> valgrind output: >>>> >>>> ==6027== Memcheck, a memory error detector >>>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>>> ==6027== Command: ex11f90 >>>> ==6027== >>>> ==6027== Invalid read of size 8 >>>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>> ==6027== >>>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>> ==6027== >>>> ==6027== Invalid read of size 8 >>>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>>> ==6027== by 0xEE1CC2F: ??? >>>> ==6027== by 0xEE5A0AF: ??? >>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>> ==6027== by 0x7FEFFFC13: ??? >>>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>>> ==6027== >>>> >>>> >>>> >>>>> >>>>> >>>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>>> >>>>> type field >>>>> >>>>> real u,v,w,p >>>>> >>>>> end type field >>>>> >>>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>>> >>>>> Also: >>>>> >>>>> Vec field_local,field_global >>>>> >>>>> call DMDACreate3d with dof = 4 >>>>> >>>>> call DMCreateGlobalVector(da, field_local,ierr) >>>>> >>>>> call DMCreateLocalVector(da,field_global,ierr) >>>>> >>>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>>> >>>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>>> >>>>> do k = zs, zs + zm - 1 >>>>> >>>>> do j = ys, ys + ym -1 >>>>> >>>>> do i = xs, xs + xm - 1 >>>>> >>>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>>> >>>>> end do >>>>> >>>>> end do >>>>> >>>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>>> >>>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>>> >>>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>>> >>>>> call DMRestoreLocalVector(da,field_local,ierr) >>>>> >>>>> Is this the correct way? >>>>> >>>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>>> >>>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>>> >>>>> For b, do I duplicate a Vec similar to field_local? >>>>> >>>>> What about matrix A? Do I use the MatSetValuesStencil? >>>>> >>>>> Yes >>>>> >>>>> >>>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>>> >>>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener From zonexo at gmail.com Mon Jun 18 22:34:12 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 18 Jun 2012 23:34:12 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> <4FDFE9CB.1040303@gmail.com> Message-ID: <4FDFF333.6060306@gmail.com> On 18/6/2012 10:58 PM, Barry Smith wrote: > On Jun 18, 2012, at 9:54 PM, TAY wee-beng wrote: > >> On 18/6/2012 10:36 PM, Barry Smith wrote: >>> Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f >>> >>> Barry >> ex19 and ex5f both run w/o problem. >> >> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex19 >> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >> Number of SNES iterations = 2 >> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex19 >> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >> Number of SNES iterations = 2 >> >> [wtay at hpc12:tutorials]$ ./ex5f >> Number of SNES iterations = 4 >> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex5f >> Number of SNES iterations = 4 >> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex5f >> Number of SNES iterations = 4 >> [wtay at hpc12:tutorials]$ >> >> The problem seems to lie in DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90. I have reported this problem before when I convert ex29 from C to Fortran. It works in Vs2008 windows but not in linux. I wonder if it has some bugs or compatibility problem. > Then likely it is an Intel compiler bug. Does this happen with the gnu gfortran compiler combined with gcc as the c compiler? > > Barry I have not tested. I'm also using intel fortran with VS2008 in windows 7 64bit. Do I have to re-build PETSc to test with gcc and gfortran? Thanks. >>> On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: >>> >>>> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>>>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>>>> >>>>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>>>> >>>>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>>>> >>>>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>>>> >>>>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>>>> I tried to build and run ex11f90 in linux. However, it's not working: >>>>> >>>>> I have attached the run and valgrind output: >>>>> >>>>> It appears to be a problem with your MPI. >>>>> >>>>> Matt >>>> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>>>> run output: >>>>> >>>>> [wtay at hpc12:tutorials]$ ./ex11f90 >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>> [0]PETSC ERROR: is given. >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Signal received! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>>>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>>>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>> -------------------------------------------------------------------------- >>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>> with errorcode 59. >>>>> >>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>> You may or may not see output from other processes, depending on >>>>> exactly when Open MPI kills them. >>>>> -------------------------------------------------------------------------- >>>>> >>>>> valgrind output: >>>>> >>>>> ==6027== Memcheck, a memory error detector >>>>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>>>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>>>> ==6027== Command: ex11f90 >>>>> ==6027== >>>>> ==6027== Invalid read of size 8 >>>>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>>>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>>>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>>>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>>>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>>>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>>>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>>>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>> ==6027== >>>>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>>>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>>>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>>>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>>>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>>>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>>>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>>>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>>>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>>>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>>>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>>>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>>>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>> ==6027== >>>>> ==6027== Invalid read of size 8 >>>>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>>>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>>>> ==6027== by 0xEE1CC2F: ??? >>>>> ==6027== by 0xEE5A0AF: ??? >>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>> ==6027== by 0x7FEFFFC13: ??? >>>>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>>>> ==6027== >>>>> >>>>> >>>>> >>>>>> >>>>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>>>> >>>>>> type field >>>>>> >>>>>> real u,v,w,p >>>>>> >>>>>> end type field >>>>>> >>>>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>>>> >>>>>> Also: >>>>>> >>>>>> Vec field_local,field_global >>>>>> >>>>>> call DMDACreate3d with dof = 4 >>>>>> >>>>>> call DMCreateGlobalVector(da, field_local,ierr) >>>>>> >>>>>> call DMCreateLocalVector(da,field_global,ierr) >>>>>> >>>>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>>>> >>>>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>>>> >>>>>> do k = zs, zs + zm - 1 >>>>>> >>>>>> do j = ys, ys + ym -1 >>>>>> >>>>>> do i = xs, xs + xm - 1 >>>>>> >>>>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>>>> >>>>>> end do >>>>>> >>>>>> end do >>>>>> >>>>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>>>> >>>>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>> >>>>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>> >>>>>> call DMRestoreLocalVector(da,field_local,ierr) >>>>>> >>>>>> Is this the correct way? >>>>>> >>>>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>>>> >>>>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>>>> >>>>>> For b, do I duplicate a Vec similar to field_local? >>>>>> >>>>>> What about matrix A? Do I use the MatSetValuesStencil? >>>>>> >>>>>> Yes >>>>>> >>>>>> >>>>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>>>> >>>>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>> -- Norbert Wiener From bsmith at mcs.anl.gov Mon Jun 18 22:36:28 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Jun 2012 22:36:28 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDFF333.6060306@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> <4FDFE9CB.1040303@gmail.com> <4FDFF333.6060306@gmail.com> Message-ID: <0EAADDAB-AAFF-4FE1-A46F-9687F290D238@mcs.anl.gov> On Jun 18, 2012, at 10:34 PM, TAY wee-beng wrote: > > On 18/6/2012 10:58 PM, Barry Smith wrote: >> On Jun 18, 2012, at 9:54 PM, TAY wee-beng wrote: >> >>> On 18/6/2012 10:36 PM, Barry Smith wrote: >>>> Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f >>>> >>>> Barry >>> ex19 and ex5f both run w/o problem. >>> >>> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex19 >>> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >>> Number of SNES iterations = 2 >>> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex19 >>> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >>> Number of SNES iterations = 2 >>> >>> [wtay at hpc12:tutorials]$ ./ex5f >>> Number of SNES iterations = 4 >>> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex5f >>> Number of SNES iterations = 4 >>> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex5f >>> Number of SNES iterations = 4 >>> [wtay at hpc12:tutorials]$ >>> >>> The problem seems to lie in DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90. I have reported this problem before when I convert ex29 from C to Fortran. It works in Vs2008 windows but not in linux. I wonder if it has some bugs or compatibility problem. >> Then likely it is an Intel compiler bug. Does this happen with the gnu gfortran compiler combined with gcc as the c compiler? >> >> Barry > I have not tested. I'm also using intel fortran with VS2008 in windows 7 64bit. Do I have to re-build PETSc to test with gcc and gfortran? Yes. The intel compilers will be slightly different on Linux and Windows and may have bug on linux but not on windows. Barry > > Thanks. >>>> On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: >>>> >>>>> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>>>>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>>>>> >>>>>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>>>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>>>>> >>>>>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>>>>> >>>>>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>>>>> >>>>>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>>>>> I tried to build and run ex11f90 in linux. However, it's not working: >>>>>> >>>>>> I have attached the run and valgrind output: >>>>>> >>>>>> It appears to be a problem with your MPI. >>>>>> >>>>>> Matt >>>>> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>>>>> run output: >>>>>> >>>>>> [wtay at hpc12:tutorials]$ ./ex11f90 >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>> [0]PETSC ERROR: is given. >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [0]PETSC ERROR: Signal received! >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>>>>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>>>>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>>>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>> -------------------------------------------------------------------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>> with errorcode 59. >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> -------------------------------------------------------------------------- >>>>>> >>>>>> valgrind output: >>>>>> >>>>>> ==6027== Memcheck, a memory error detector >>>>>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>>>>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>>>>> ==6027== Command: ex11f90 >>>>>> ==6027== >>>>>> ==6027== Invalid read of size 8 >>>>>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>>>>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>>>>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>>>>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>>>>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>>>>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>>>>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>>>>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>> ==6027== >>>>>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>>>>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>>>>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>>>>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>>>>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>>>>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>>>>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>>>>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>>>>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>>>>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>>>>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>>>>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>>>>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>> ==6027== >>>>>> ==6027== Invalid read of size 8 >>>>>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>>>>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>>>>> ==6027== by 0xEE1CC2F: ??? >>>>>> ==6027== by 0xEE5A0AF: ??? >>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>> ==6027== by 0x7FEFFFC13: ??? >>>>>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>>>>> ==6027== >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>>>>> >>>>>>> type field >>>>>>> >>>>>>> real u,v,w,p >>>>>>> >>>>>>> end type field >>>>>>> >>>>>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>>>>> >>>>>>> Also: >>>>>>> >>>>>>> Vec field_local,field_global >>>>>>> >>>>>>> call DMDACreate3d with dof = 4 >>>>>>> >>>>>>> call DMCreateGlobalVector(da, field_local,ierr) >>>>>>> >>>>>>> call DMCreateLocalVector(da,field_global,ierr) >>>>>>> >>>>>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>>>>> >>>>>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>>>>> >>>>>>> do k = zs, zs + zm - 1 >>>>>>> >>>>>>> do j = ys, ys + ym -1 >>>>>>> >>>>>>> do i = xs, xs + xm - 1 >>>>>>> >>>>>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>>>>> >>>>>>> end do >>>>>>> >>>>>>> end do >>>>>>> >>>>>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>>>>> >>>>>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>> >>>>>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>> >>>>>>> call DMRestoreLocalVector(da,field_local,ierr) >>>>>>> >>>>>>> Is this the correct way? >>>>>>> >>>>>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>>>>> >>>>>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>>>>> >>>>>>> For b, do I duplicate a Vec similar to field_local? >>>>>>> >>>>>>> What about matrix A? Do I use the MatSetValuesStencil? >>>>>>> >>>>>>> Yes >>>>>>> >>>>>>> >>>>>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>>>>> >>>>>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>> -- Norbert Wiener From zonexo at gmail.com Mon Jun 18 23:04:43 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 19 Jun 2012 00:04:43 -0400 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <0EAADDAB-AAFF-4FE1-A46F-9687F290D238@mcs.anl.gov> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> <4FDFE9CB.1040303@gmail.com> <4FDFF333.6060306@gmail.com> <0EAADDAB-AAFF-4FE1-A46F-9687F290D238@mcs.anl.gov> Message-ID: <4FDFFA5B.5060308@gmail.com> On 18/6/2012 11:36 PM, Barry Smith wrote: > On Jun 18, 2012, at 10:34 PM, TAY wee-beng wrote: > >> On 18/6/2012 10:58 PM, Barry Smith wrote: >>> On Jun 18, 2012, at 9:54 PM, TAY wee-beng wrote: >>> >>>> On 18/6/2012 10:36 PM, Barry Smith wrote: >>>>> Do PETSc examples run on this system. cd src/snes/examples/tutorials; make ex19 ; then run ex19 on a bunch of nodes of the cluster, does it run correctly? Repeat with ex5f >>>>> >>>>> Barry >>>> ex19 and ex5f both run w/o problem. >>>> >>>> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex19 >>>> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >>>> Number of SNES iterations = 2 >>>> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex19 >>>> lid velocity = 0.0625, prandtl # = 1, grashof # = 1 >>>> Number of SNES iterations = 2 >>>> >>>> [wtay at hpc12:tutorials]$ ./ex5f >>>> Number of SNES iterations = 4 >>>> [wtay at hpc12:tutorials]$ mpiexec -np 2 ./ex5f >>>> Number of SNES iterations = 4 >>>> [wtay at hpc12:tutorials]$ mpiexec -np 4 ./ex5f >>>> Number of SNES iterations = 4 >>>> [wtay at hpc12:tutorials]$ >>>> >>>> The problem seems to lie in DMDAVecGetArrayF90 and DMDAVecRestoreArrayF90. I have reported this problem before when I convert ex29 from C to Fortran. It works in Vs2008 windows but not in linux. I wonder if it has some bugs or compatibility problem. >>> Then likely it is an Intel compiler bug. Does this happen with the gnu gfortran compiler combined with gcc as the c compiler? >>> >>> Barry >> I have not tested. I'm also using intel fortran with VS2008 in windows 7 64bit. Do I have to re-build PETSc to test with gcc and gfortran? > Yes. The intel compilers will be slightly different on Linux and Windows and may have bug on linux but not on windows. > > Barry Yup, gfortran + gcc works. So do I submit a bug report to Intel? I've heard that gfortran is slower compared to ifort. Is this true? > >> Thanks. >>>>> On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: >>>>> >>>>>> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>>>>>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>>>>>> >>>>>>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>>>>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>>>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>>>>>> >>>>>>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>>>>>> >>>>>>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>>>>>> I tried to build and run ex11f90 in linux. However, it's not working: >>>>>>> >>>>>>> I have attached the run and valgrind output: >>>>>>> >>>>>>> It appears to be a problem with your MPI. >>>>>>> >>>>>>> Matt >>>>>> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>>>>>> run output: >>>>>>> >>>>>>> [wtay at hpc12:tutorials]$ ./ex11f90 >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>> [0]PETSC ERROR: is given. >>>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>> [0]PETSC ERROR: Signal received! >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>>>>>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>>>>>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>>>>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>> -------------------------------------------------------------------------- >>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>> with errorcode 59. >>>>>>> >>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>> You may or may not see output from other processes, depending on >>>>>>> exactly when Open MPI kills them. >>>>>>> -------------------------------------------------------------------------- >>>>>>> >>>>>>> valgrind output: >>>>>>> >>>>>>> ==6027== Memcheck, a memory error detector >>>>>>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>>>>>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>>>>>> ==6027== Command: ex11f90 >>>>>>> ==6027== >>>>>>> ==6027== Invalid read of size 8 >>>>>>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>>>>>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>>>>>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>>>>>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>>>>>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>>>>>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>>>>>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>>>>>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>> ==6027== >>>>>>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>>>>>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>>>>>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>>>>>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>>>>>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>>>>>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>>>>>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>>>>>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>>>>>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>>>>>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>>>>>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>>>>>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>>>>>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>> ==6027== >>>>>>> ==6027== Invalid read of size 8 >>>>>>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>>>>>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>>>>>> ==6027== by 0xEE1CC2F: ??? >>>>>>> ==6027== by 0xEE5A0AF: ??? >>>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>> ==6027== by 0x7FEFFFC13: ??? >>>>>>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>>>>>> ==6027== >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>>>>>> >>>>>>>> type field >>>>>>>> >>>>>>>> real u,v,w,p >>>>>>>> >>>>>>>> end type field >>>>>>>> >>>>>>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>>>>>> >>>>>>>> Also: >>>>>>>> >>>>>>>> Vec field_local,field_global >>>>>>>> >>>>>>>> call DMDACreate3d with dof = 4 >>>>>>>> >>>>>>>> call DMCreateGlobalVector(da, field_local,ierr) >>>>>>>> >>>>>>>> call DMCreateLocalVector(da,field_global,ierr) >>>>>>>> >>>>>>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>>>>>> >>>>>>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>>>>>> >>>>>>>> do k = zs, zs + zm - 1 >>>>>>>> >>>>>>>> do j = ys, ys + ym -1 >>>>>>>> >>>>>>>> do i = xs, xs + xm - 1 >>>>>>>> >>>>>>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>>>>>> >>>>>>>> end do >>>>>>>> >>>>>>>> end do >>>>>>>> >>>>>>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>>>>>> >>>>>>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>>> >>>>>>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>>> >>>>>>>> call DMRestoreLocalVector(da,field_local,ierr) >>>>>>>> >>>>>>>> Is this the correct way? >>>>>>>> >>>>>>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>>>>>> >>>>>>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>>>>>> >>>>>>>> For b, do I duplicate a Vec similar to field_local? >>>>>>>> >>>>>>>> What about matrix A? Do I use the MatSetValuesStencil? >>>>>>>> >>>>>>>> Yes >>>>>>>> >>>>>>>> >>>>>>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>>>>>> >>>>>>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>> -- Norbert Wiener From bsmith at mcs.anl.gov Mon Jun 18 23:08:10 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 18 Jun 2012 23:08:10 -0500 Subject: [petsc-users] DOF in DMDACreate3d and staggered grid In-Reply-To: <4FDFFA5B.5060308@gmail.com> References: <4FDC8602.5060304@gmail.com> <4FDDCD0C.5080302@gmail.com> <4FDFD62F.3030702@gmail.com> <4FDFE21B.30903@gmail.com> <0D40A1C9-3A07-4678-84E0-16950B155868@mcs.anl.gov> <4FDFE9CB.1040303@gmail.com> <4FDFF333.6060306@gmail.com> <0EAADDAB-AAFF-4FE1-A46F-9687F290D238@mcs.anl.gov> <4FDFFA5B.5060308@gmail.com> Message-ID: <9987C237-F7D4-4755-BFB4-6804D1466C5F@mcs.anl.gov> On Jun 18, 2012, at 11:04 PM, TAY wee-beng wrote: > > On 18/6/2012 11:36 PM, Barry Smith wrote: >> On Jun 18, 2012, at 10:34 PM, TAY wee-beng wrote: >> >> >> Barry > > Yup, gfortran + gcc works. So do I submit a bug report to Intel? > > I've heard that gfortran is slower compared to ifort. Is this true? Run something YOU care about and compare the times. Likely Intel won't be much faster and besides being much faster and NOT working is not very helpful. Barry > > >> >>> Thanks. >>>>>> On Jun 18, 2012, at 9:21 PM, TAY wee-beng wrote: >>>>>> >>>>>>> On 18/6/2012 9:52 PM, Matthew Knepley wrote: >>>>>>>> On Mon, Jun 18, 2012 at 7:30 PM, TAY wee-beng wrote: >>>>>>>> >>>>>>>> On 17/6/2012 2:33 PM, Jed Brown wrote: >>>>>>>>> On Sun, Jun 17, 2012 at 7:26 AM, TAY wee-beng wrote: >>>>>>>>> >>>>>>>>> On 16/6/2012 9:24 AM, Jed Brown wrote: >>>>>>>>>> It depends how you want to solve the problem. I usually group all dofs together. There is a 2D Stokes+Thermodynamics example in SNES ex30 (or 31?). >>>>>>>>>> >>>>>>>>> I tried to understand ex30. I have some questions since it's in C and I'm using to Fortran programming. >>>>>>>>> >>>>>>>>> This looks about right, see src/dm/examples/tutorials/ex11f90.F. >>>>>>>> I tried to build and run ex11f90 in linux. However, it's not working: >>>>>>>> >>>>>>>> I have attached the run and valgrind output: >>>>>>>> >>>>>>>> It appears to be a problem with your MPI. >>>>>>>> >>>>>>>> Matt >>>>>>> Is there anyway to solve this problem? I'm running on a cluster using openmpi 1.5.3 and I've no admin rights. >>>>>>>> run output: >>>>>>>> >>>>>>>> [wtay at hpc12:tutorials]$ ./ex11f90 >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>> [0]PETSC ERROR: likely location of problem given in stack below >>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>> [0]PETSC ERROR: is given. >>>>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>>>> [0]PETSC ERROR: Signal received! >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: Petsc Development HG revision: f3b998b41b349e16d47fe42b0e223d3462737e05 HG Date: Fri Jun 15 17:50:32 2012 -0500 >>>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: ./ex11f90 on a petsc-3.3 named hpc12 by wtay Tue Jun 19 03:22:52 2012 >>>>>>>> [0]PETSC ERROR: Libraries linked from /home/wtay/Lib/petsc-3.3-dev_shared_debug/lib >>>>>>>> [0]PETSC ERROR: Configure run at Sun Jun 17 16:51:29 2012 >>>>>>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=1 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_debug --known-mpi-shared=1 --with-shared-libraries >>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>> with errorcode 59. >>>>>>>> >>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>> You may or may not see output from other processes, depending on >>>>>>>> exactly when Open MPI kills them. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> >>>>>>>> valgrind output: >>>>>>>> >>>>>>>> ==6027== Memcheck, a memory error detector >>>>>>>> ==6027== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. >>>>>>>> ==6027== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info >>>>>>>> ==6027== Command: ex11f90 >>>>>>>> ==6027== >>>>>>>> ==6027== Invalid read of size 8 >>>>>>>> ==6027== at 0xA60148D: _wordcopy_fwd_dest_aligned (in /lib64/libc-2.12.so) >>>>>>>> ==6027== by 0xA5FB11D: __GI_memmove (in /lib64/libc-2.12.so) >>>>>>>> ==6027== by 0xA6027DB: argz_insert (in /lib64/libc-2.12.so) >>>>>>>> ==6027== by 0x95CAF25: lt_argz_insert (ltdl.c:1679) >>>>>>>> ==6027== by 0x95CB7D0: foreachfile_callback (ltdl.c:1718) >>>>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>>>> ==6027== Address 0xb39d9a8 is 40 bytes inside a block of size 43 alloc'd >>>>>>>> ==6027== at 0x4C267BA: malloc (vg_replace_malloc.c:263) >>>>>>>> ==6027== by 0x95CA358: lt__malloc (lt__alloc.c:54) >>>>>>>> ==6027== by 0x95CB751: foreachfile_callback (ltdl.c:1764) >>>>>>>> ==6027== by 0x95CB4F1: foreach_dirinpath (ltdl.c:710) >>>>>>>> ==6027== by 0x95CB580: lt_dlforeachfile (ltdl.c:1865) >>>>>>>> ==6027== by 0x95DB999: mca_base_component_find (mca_base_component_find.c:301) >>>>>>>> ==6027== by 0x95DC4B0: mca_base_components_open (mca_base_components_open.c:128) >>>>>>>> ==6027== by 0x95F7CE7: opal_paffinity_base_open (paffinity_base_open.c:112) >>>>>>>> ==6027== by 0x95C39FE: opal_init (opal_init.c:307) >>>>>>>> ==6027== by 0x95815A4: orte_init (orte_init.c:78) >>>>>>>> ==6027== by 0x95432AE: ompi_mpi_init (ompi_mpi_init.c:350) >>>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>>> ==6027== >>>>>>>> ==6027== Syscall param writev(vector[...]) points to uninitialised byte(s) >>>>>>>> ==6027== at 0xA65692B: writev (in /lib64/libc-2.12.so) >>>>>>>> ==6027== by 0xC9A2C56: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249) >>>>>>>> ==6027== by 0xC9A417C: mca_oob_tcp_peer_send (oob_tcp_peer.c:204) >>>>>>>> ==6027== by 0xC9A67FC: mca_oob_tcp_send_nb (oob_tcp_send.c:167) >>>>>>>> ==6027== by 0xC3953B5: orte_rml_oob_send (rml_oob_send.c:136) >>>>>>>> ==6027== by 0xC3955FF: orte_rml_oob_send_buffer (rml_oob_send.c:270) >>>>>>>> ==6027== by 0xCDB1E87: modex (grpcomm_bad_module.c:573) >>>>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>>>> ==6027== Address 0xed30cd1 is 161 bytes inside a block of size 256 alloc'd >>>>>>>> ==6027== at 0x4C268B2: realloc (vg_replace_malloc.c:632) >>>>>>>> ==6027== by 0x95C4F22: opal_dss_buffer_extend (dss_internal_functions.c:63) >>>>>>>> ==6027== by 0x95C5A64: opal_dss_copy_payload (dss_load_unload.c:164) >>>>>>>> ==6027== by 0x95A1246: orte_grpcomm_base_pack_modex_entries (grpcomm_base_modex.c:861) >>>>>>>> ==6027== by 0xCDB1E3C: modex (grpcomm_bad_module.c:563) >>>>>>>> ==6027== by 0x95436F1: ompi_mpi_init (ompi_mpi_init.c:682) >>>>>>>> ==6027== by 0x955938F: PMPI_Init (pinit.c:84) >>>>>>>> ==6027== by 0x8AB4FF4: MPI_INIT (pinit_f.c:75) >>>>>>>> ==6027== by 0x50ED97E: petscinitialize_ (zstart.c:299) >>>>>>>> ==6027== by 0x40881C: MAIN__ (ex11f90.F:43) >>>>>>>> ==6027== by 0x4087AB: main (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>>> ==6027== >>>>>>>> ==6027== Invalid read of size 8 >>>>>>>> ==6027== at 0x50FC6D8: vecview_ (zvectorf.c:56) >>>>>>>> ==6027== by 0x408A05: MAIN__ (ex11f90.F:56) >>>>>>>> ==6027== by 0xEE1CC2F: ??? >>>>>>>> ==6027== by 0xEE5A0AF: ??? >>>>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>>> ==6027== by 0x4962FF: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>>> ==6027== by 0x6F5C9F: ??? (in /home/wtay/Codes/petsc-dev/src/dm/examples/tutorials/ex11f90) >>>>>>>> ==6027== by 0x7FEFFFC13: ??? >>>>>>>> ==6027== Address 0xfffffffffffffeb8 is not stack'd, malloc'd or (recently) free'd >>>>>>>> ==6027== >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Supposed I have a field - u,v,w,p, so in order to use them, I do the following: >>>>>>>>> >>>>>>>>> type field >>>>>>>>> >>>>>>>>> real u,v,w,p >>>>>>>>> >>>>>>>>> end type field >>>>>>>>> >>>>>>>>> type(field), pointer :: field1(:,:,:) -> make a derived variable >>>>>>>>> >>>>>>>>> Also: >>>>>>>>> >>>>>>>>> Vec field_local,field_global >>>>>>>>> >>>>>>>>> call DMDACreate3d with dof = 4 >>>>>>>>> >>>>>>>>> call DMCreateGlobalVector(da, field_local,ierr) >>>>>>>>> >>>>>>>>> call DMCreateLocalVector(da,field_global,ierr) >>>>>>>>> >>>>>>>>> call DMGetLocalVector(da, field_local,ierr) -> To insert values >>>>>>>>> >>>>>>>>> call DMDAVecGetArrayF90(da, field_local,field1,ierr) >>>>>>>>> >>>>>>>>> do k = zs, zs + zm - 1 >>>>>>>>> >>>>>>>>> do j = ys, ys + ym -1 >>>>>>>>> >>>>>>>>> do i = xs, xs + xm - 1 >>>>>>>>> >>>>>>>>> field1(i,j,k)%u = ... -> evaluate u,v,w,p etc >>>>>>>>> >>>>>>>>> end do >>>>>>>>> >>>>>>>>> end do >>>>>>>>> >>>>>>>>> call DMDAVecRestoreArrayF90(da,field_local,field1,ierr) >>>>>>>>> >>>>>>>>> call DMLocalToGlobalBegin(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>>>> >>>>>>>>> call DMLocalToGlobalEnd(da,field_local,INSERT_VALUES,field_global,ierr) >>>>>>>>> >>>>>>>>> call DMRestoreLocalVector(da,field_local,ierr) >>>>>>>>> >>>>>>>>> Is this the correct way? >>>>>>>>> >>>>>>>>> Also, supposed I now want to solve my u,v,w momentum eqns. Although they're not coupled together, I believe it's faster if I assemble them into 1 big matrix. >>>>>>>>> >>>>>>>>> So for Ax = b, x = (field(1,1,1)%u,field(1,1,1)%v,field(1,1,1)%w,field(2,1,1)%u.... ) >>>>>>>>> >>>>>>>>> For b, do I duplicate a Vec similar to field_local? >>>>>>>>> >>>>>>>>> What about matrix A? Do I use the MatSetValuesStencil? >>>>>>>>> >>>>>>>>> Yes >>>>>>>>> >>>>>>>>> >>>>>>>>> Lastly, the type field contains u,v,w and p. However, I'm only solving u,v,w. Do I have to skip some values or use identity matrix to solve it? >>>>>>>>> >>>>>>>>> Why not make a field that contains only u,v,w. I don't see what you're trying to do. >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>> -- Norbert Wiener From fd.kong at siat.ac.cn Tue Jun 19 01:17:59 2012 From: fd.kong at siat.ac.cn (=?ISO-8859-1?B?ZmRrb25n?=) Date: Tue, 19 Jun 2012 14:17:59 +0800 Subject: [petsc-users] Can c++ SieveMesh support 64-bit? Message-ID: Hi Matt, Thank you for your help! >On Mon, Jun 18, 2012 at 7:04 PM, fdkong wrote: >> Hi Mat, >> >> I have been developing some codes based-on c++ sievemesh (your old version >> code). I want to know the following questions: >> >> (1) You have added a new c sievemesh. Will the old c++ version be remove >> in future? >> >Yes, but not soon. However, I recommend making a plan for >switching, since >you get so many benefits >including much better solver integration. Take a look at SNES ex62. The c++ version is very different from c version. All of my codes have been built on the top of c++ version. I directly use inner object ALE::IMesh. Thus, switching to C version may be difficult. Could you give me some suggestions how to transfer my code from c++ to c? >> (2) Have you ever made some tests on 64-bit computer? Whether c++ >> sieveMesh can run on 64-bit computer? I want to use 64-bit integer. >> >It will work on a 64-bit computer. Both version use PetscInt. Thank you, I will make some tests on 64-bit computer. > Matt >> Regards, >> ** >> ------------------ >> Fande Kong >> ShenZhen Institutes of Advanced Technology >> Chinese Academy of Sciences >> ** -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Tue Jun 19 01:55:37 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Tue, 19 Jun 2012 06:55:37 +0000 Subject: [petsc-users] function name and meaning of -pc_fieldsplit_schur_factorization_type Message-ID: > > > It looks like PCFieldSplitSetSchurFactorizationType() is missing. > > > > Would be nice to have. > > > I don't know if I ever pointed this out to you. It's in 3.3. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/6d8eec > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ead058 Thanks for pointing this out Jed, I'll upgrade to 3.3. Chris From knepley at gmail.com Tue Jun 19 08:50:41 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 07:50:41 -0600 Subject: [petsc-users] Can c++ SieveMesh support 64-bit? In-Reply-To: References: Message-ID: On Tue, Jun 19, 2012 at 12:17 AM, fdkong wrote: > Hi Matt, > > Thank you for your help! > > **>On Mon, Jun 18, 2012 at 7:04 PM, fdkong wrote: > > >> Hi Mat, > >> > >> I have been developing some codes based-on c++ sievemesh (your old > version > >> code). I want to know the following questions: > >> > >> (1) You have added a new c sievemesh. Will the old c++ version be remove > >> in future? > >> > > >Yes, but not soon. However, I recommend making a plan for >switching, > since > >you get so many benefits > >including much better solver integration. Take a look at SNES ex62. > > The c++ version is very different from c version. All of my codes have > been built on the top of c++ version. I directly use inner object > ALE::IMesh. Thus, switching to C version may be > difficult. Could you give me some suggestions how to transfer my code from > c++ to c? > Hopefully its not that hard. The interface is the same. Maybe you could ask me questions if you encounter a problem. Matt > >> (2) Have you ever made some tests on 64-bit computer? Whether c++ > >> sieveMesh can run on 64-bit computer? I want to use 64-bit integer. > >> > > ****>It will work on a 64-bit computer. Both version use PetscInt. > > ****Thank you, I will make some tests on 64-bit computer. > > ** > ** > Matt > > > >> Regards, > >> ** > >> ------------------ > >> Fande Kong > >> ShenZhen Institutes of Advanced Technology > >> Chinese Academy of Sciences > >> ** > > ** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Tue Jun 19 11:06:52 2012 From: popov at uni-mainz.de (Anton Popov) Date: Tue, 19 Jun 2012 18:06:52 +0200 Subject: [petsc-users] MatSetOption bug? Message-ID: <4FE0A39C.8030808@uni-mainz.de> Hi petsc team! I've recently upgraded to 3.3, and currently experience problems. One of them follows. If I compile this simple code: #include "petsc.h" #undef __FUNCT__ #define __FUNCT__ "main" int main( int argc,char **argv ) { Mat A; PetscErrorCode ierr; PetscInt n_rows_Local=100; PetscInt n_cols_Local=100; ierr = PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); CHKERRQ(ierr); ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr); ierr = MatSetSizes(A, n_rows_Local, n_cols_Local, PETSC_DETERMINE, PETSC_DETERMINE); CHKERRQ(ierr); ierr = MatSetType(A, MATAIJ); CHKERRQ(ierr); ierr = MatSetOption(A, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); CHKERRQ(ierr); ierr = PetscFinalize(); CHKERRQ(ierr); PetscFunctionReturn(0); } and run it in parallel (mpiexec -np 2 ./example) it gives me errors like this: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Null argument, when expecting valid pointer! [0]PETSC ERROR: Null Object: Parameter # 1! [0]PETSC ERROR: ------------------------------------------------------------------------ Commenting the line with "MatSetOption" removes the error, but doesn't solve the problem, because I actually need to KEEP_NONZERO_PATTERN Running the code sequentially, also vanishes the error. Please check whether it's a bug in petsc-3.3, or I should set this option in a different way. Thanks, Anton Popov From margarita.satraki at gmail.com Tue Jun 19 11:33:21 2012 From: margarita.satraki at gmail.com (Margarita Satraki) Date: Tue, 19 Jun 2012 17:33:21 +0100 Subject: [petsc-users] PCKSP Message-ID: Hello, I have difficulty understanding how PCKSP works. From: http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf I understand that instead of using preconditioners, it uses Krylov methods for the ''inner solvers''. What are the ''inner solvers''? Is there some kind of a subsystem that is solved instead of applying a preconditioner? Many thanks, Margarita -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 19 11:49:35 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 10:49:35 -0600 Subject: [petsc-users] MatSetOption bug? In-Reply-To: <4FE0A39C.8030808@uni-mainz.de> References: <4FE0A39C.8030808@uni-mainz.de> Message-ID: On Tue, Jun 19, 2012 at 10:06 AM, Anton Popov wrote: > Hi petsc team! > I've recently upgraded to 3.3, and currently experience problems. One of > them follows. > If I compile this simple code: > In 3.3, we now require that MatSetUp() is called before messing with the storage scheme. We should have been more specific in Changes. Moreover, this should be a better error message. Will fix, and see below. > #include "petsc.h" > #undef __FUNCT__ > #define __FUNCT__ "main" > int main( int argc,char **argv ) > { > Mat A; > PetscErrorCode ierr; > PetscInt n_rows_Local=100; > PetscInt n_cols_Local=100; > ierr = PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); > CHKERRQ(ierr); > ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr); > ierr = MatSetSizes(A, n_rows_Local, n_cols_Local, PETSC_DETERMINE, > PETSC_DETERMINE); CHKERRQ(ierr); > ierr = MatSetType(A, MATAIJ); CHKERRQ(ierr); > ierr = MatSetUp(A);CHKERRQ(ierr); Matt > ierr = MatSetOption(A, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); > CHKERRQ(ierr); > ierr = PetscFinalize(); CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > and run it in parallel (mpiexec -np 2 ./example) it gives me errors like > this: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > > Commenting the line with "MatSetOption" removes the error, but doesn't > solve the problem, because I actually need to KEEP_NONZERO_PATTERN > > Running the code sequentially, also vanishes the error. > > Please check whether it's a bug in petsc-3.3, or I should set this option > in a different way. > > Thanks, > > Anton Popov > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bishtg at ornl.gov Tue Jun 19 11:54:12 2012 From: bishtg at ornl.gov (Bisht, Gautam) Date: Tue, 19 Jun 2012 12:54:12 -0400 Subject: [petsc-users] 64-bit Fortran binding for external graph partitioning packages Message-ID: <7D7237F5271FF342A4E61F5AE0FA9716622D504199@EXCHMBB.ornl.gov> Hi, Out of the four external graph partitioning packages supported by PETSc: Chaco, ParMETIS, Party, and PTScotch, I was wondering if anyone could comment which of these graph partioning support 64-bit with Fortran? My understanding is that ParMETIS does not support 64-bit with Fortran. Thanks, -Gautam. From knepley at gmail.com Tue Jun 19 11:54:46 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 10:54:46 -0600 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < margarita.satraki at gmail.com> wrote: > Hello, > > I have difficulty understanding how PCKSP works. > > From: > > http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf > I understand that instead of using preconditioners, it uses Krylov methods > for the ''inner solvers''. > > What are the ''inner solvers''? Is there some kind of a subsystem that is > solved instead of applying a preconditioner? > Nope, its jsut like a PC: M^{-1} A x = M^{-1} b where now M^{-1} instead of being an LU solve, for instance, is a Krylov solve. Matt > Many thanks, > > Margarita > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From margarita.satraki at gmail.com Tue Jun 19 12:41:04 2012 From: margarita.satraki at gmail.com (Margarita Satraki) Date: Tue, 19 Jun 2012 18:41:04 +0100 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: If I understand correctly: For a system: M^{-1} A x = M^{-1} b we don't need to multiply M^{-1} A explicitly, but we solve M w = v whenever needed. So the Krylov method is used in order to solve that system, or equivalently to compute the vector M^{-1} v? On 19 June 2012 17:54, Matthew Knepley wrote: > On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < > margarita.satraki at gmail.com> wrote: > >> Hello, >> >> I have difficulty understanding how PCKSP works. >> >> From: >> >> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >> I understand that instead of using preconditioners, it uses Krylov >> methods for the ''inner solvers''. >> >> What are the ''inner solvers''? Is there some kind of a subsystem that is >> solved instead of applying a preconditioner? >> > > Nope, its jsut like a PC: > > M^{-1} A x = M^{-1} b > > where now M^{-1} instead of being an LU solve, for instance, is a Krylov > solve. > > Matt > > >> Many thanks, >> >> Margarita >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 19 12:44:39 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 11:44:39 -0600 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < margarita.satraki at gmail.com> wrote: > If I understand correctly: > > For a system: > M^{-1} A x = M^{-1} b > we don't need to multiply M^{-1} A explicitly, but we solve M w = v > whenever needed. > > So the Krylov method is used in order to solve that system, or > equivalently to compute the vector M^{-1} v? > Yes Matt > > On 19 June 2012 17:54, Matthew Knepley wrote: > >> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >> margarita.satraki at gmail.com> wrote: >> >>> Hello, >>> >>> I have difficulty understanding how PCKSP works. >>> >>> From: >>> >>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>> I understand that instead of using preconditioners, it uses Krylov >>> methods for the ''inner solvers''. >>> >>> What are the ''inner solvers''? Is there some kind of a subsystem that >>> is solved instead of applying a preconditioner? >>> >> >> Nope, its jsut like a PC: >> >> M^{-1} A x = M^{-1} b >> >> where now M^{-1} instead of being an LU solve, for instance, is a Krylov >> solve. >> >> Matt >> >> >>> Many thanks, >>> >>> Margarita >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From margarita.satraki at gmail.com Tue Jun 19 12:54:21 2012 From: margarita.satraki at gmail.com (Margarita Satraki) Date: Tue, 19 Jun 2012 18:54:21 +0100 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: Thank you very much Matt. Margarita On 19 June 2012 18:44, Matthew Knepley wrote: > On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < > margarita.satraki at gmail.com> wrote: > >> If I understand correctly: >> >> For a system: >> M^{-1} A x = M^{-1} b >> we don't need to multiply M^{-1} A explicitly, but we solve M w = v >> whenever needed. >> >> So the Krylov method is used in order to solve that system, or >> equivalently to compute the vector M^{-1} v? >> > > Yes > > Matt > > >> >> On 19 June 2012 17:54, Matthew Knepley wrote: >> >>> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >>> margarita.satraki at gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I have difficulty understanding how PCKSP works. >>>> >>>> From: >>>> >>>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>>> I understand that instead of using preconditioners, it uses Krylov >>>> methods for the ''inner solvers''. >>>> >>>> What are the ''inner solvers''? Is there some kind of a subsystem that >>>> is solved instead of applying a preconditioner? >>>> >>> >>> Nope, its jsut like a PC: >>> >>> M^{-1} A x = M^{-1} b >>> >>> where now M^{-1} instead of being an LU solve, for instance, is a Krylov >>> solve. >>> >>> Matt >>> >>> >>>> Many thanks, >>>> >>>> Margarita >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 19 13:00:28 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 12:00:28 -0600 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: On Tue, Jun 19, 2012 at 11:54 AM, Margarita Satraki < margarita.satraki at gmail.com> wrote: > Thank you very much Matt. > I will just point out that this is unlikely to ever be what you want. Do you have a particular solver configuration in mind? Matt > Margarita > > On 19 June 2012 18:44, Matthew Knepley wrote: > >> On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < >> margarita.satraki at gmail.com> wrote: >> >>> If I understand correctly: >>> >>> For a system: >>> M^{-1} A x = M^{-1} b >>> we don't need to multiply M^{-1} A explicitly, but we solve M w = v >>> whenever needed. >>> >>> So the Krylov method is used in order to solve that system, or >>> equivalently to compute the vector M^{-1} v? >>> >> >> Yes >> >> Matt >> >> >>> >>> On 19 June 2012 17:54, Matthew Knepley wrote: >>> >>>> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >>>> margarita.satraki at gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have difficulty understanding how PCKSP works. >>>>> >>>>> From: >>>>> >>>>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>>>> I understand that instead of using preconditioners, it uses Krylov >>>>> methods for the ''inner solvers''. >>>>> >>>>> What are the ''inner solvers''? Is there some kind of a subsystem that >>>>> is solved instead of applying a preconditioner? >>>>> >>>> >>>> Nope, its jsut like a PC: >>>> >>>> M^{-1} A x = M^{-1} b >>>> >>>> where now M^{-1} instead of being an LU solve, for instance, is a >>>> Krylov solve. >>>> >>>> Matt >>>> >>>> >>>>> Many thanks, >>>>> >>>>> Margarita >>>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 19 13:06:23 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 19 Jun 2012 13:06:23 -0500 Subject: [petsc-users] 64-bit Fortran binding for external graph partitioning packages In-Reply-To: <7D7237F5271FF342A4E61F5AE0FA9716622D504199@EXCHMBB.ornl.gov> References: <7D7237F5271FF342A4E61F5AE0FA9716622D504199@EXCHMBB.ornl.gov> Message-ID: <0F146D97-28FE-4FF9-864A-66DD2289C45A@mcs.anl.gov> On Jun 19, 2012, at 11:54 AM, Bisht, Gautam wrote: > Hi, > > Out of the four external graph partitioning packages supported by PETSc: Chaco, ParMETIS, Party, and PTScotch, I was wondering if anyone could comment which of these graph partioning support 64-bit with Fortran? > First for clarity: you are referring to using 64 bit integers for defining the graph. All the packages support using 64 bit pointers which is a different thing. > My understanding is that ParMETIS does not support 64-bit with Fortran. Are you calling the partitioner directly from FORTRAN (that is with ParMETIS function calls) or are you using the PETSc partitioning interface? The PETSc partitioning interface should work fine in this case from FORTRAN. If you are calling ParMETIS directly yourself from FORTRAN then it is your problem you need to take up with George. Barry > > Thanks, > -Gautam. From yifli82 at gmail.com Tue Jun 19 13:09:08 2012 From: yifli82 at gmail.com (Yifei Li) Date: Tue, 19 Jun 2012 14:09:08 -0400 Subject: [petsc-users] make test failed In-Reply-To: References: Message-ID: Is there anyone who can help me with this? I got this error after I did 'make' successfully during PETSc installation. Thanks Yifei On Mon, Jun 18, 2012 at 6:21 PM, Yifei Li wrote: > Hi all, > > I got the following error when running 'make test': > > Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and > PETSC_ARCH=arch-mswin-c-debug > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near > unexpected token 'elif' > 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" ]; > then > > > Can someone help me with this? Thanks > > Yifei > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jun 19 13:10:47 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 19 Jun 2012 13:10:47 -0500 (CDT) Subject: [petsc-users] make test failed In-Reply-To: References: Message-ID: Not sure why you would get this error. You can try running tests manually. cd src/ksp/ksp/examples/tutorials make ex2 ./ex2 [if you've also built with fortran] make ex2f ./ex2f Satish On Tue, 19 Jun 2012, Yifei Li wrote: > Is there anyone who can help me with this? I got this error after I did > 'make' successfully during PETSc installation. > > Thanks > Yifei > > On Mon, Jun 18, 2012 at 6:21 PM, Yifei Li wrote: > > > Hi all, > > > > I got the following error when running 'make test': > > > > Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and > > PETSC_ARCH=arch-mswin-c-debug > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > > process > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near > > unexpected token 'elif' > > 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" ]; > > then > > > > > > Can someone help me with this? Thanks > > > > Yifei > > > From margarita.satraki at gmail.com Tue Jun 19 13:13:11 2012 From: margarita.satraki at gmail.com (Margarita Satraki) Date: Tue, 19 Jun 2012 19:13:11 +0100 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: For my problem (incompressible nonlinear elasticity) pcksp seemed to work ok with gmres and even better with fgmres. The best option was to use LU as PC but this requires a lot of memory. Why do you think that would not be a good option? Margarita On 19 June 2012 19:00, Matthew Knepley wrote: > On Tue, Jun 19, 2012 at 11:54 AM, Margarita Satraki < > margarita.satraki at gmail.com> wrote: > >> Thank you very much Matt. >> > > I will just point out that this is unlikely to ever be what you want. Do > you have a particular solver > configuration in mind? > > Matt > > >> Margarita >> >> On 19 June 2012 18:44, Matthew Knepley wrote: >> >>> On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < >>> margarita.satraki at gmail.com> wrote: >>> >>>> If I understand correctly: >>>> >>>> For a system: >>>> M^{-1} A x = M^{-1} b >>>> we don't need to multiply M^{-1} A explicitly, but we solve M w = v >>>> whenever needed. >>>> >>>> So the Krylov method is used in order to solve that system, or >>>> equivalently to compute the vector M^{-1} v? >>>> >>> >>> Yes >>> >>> Matt >>> >>> >>>> >>>> On 19 June 2012 17:54, Matthew Knepley wrote: >>>> >>>>> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >>>>> margarita.satraki at gmail.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have difficulty understanding how PCKSP works. >>>>>> >>>>>> From: >>>>>> >>>>>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>>>>> I understand that instead of using preconditioners, it uses Krylov >>>>>> methods for the ''inner solvers''. >>>>>> >>>>>> What are the ''inner solvers''? Is there some kind of a subsystem >>>>>> that is solved instead of applying a preconditioner? >>>>>> >>>>> >>>>> Nope, its jsut like a PC: >>>>> >>>>> M^{-1} A x = M^{-1} b >>>>> >>>>> where now M^{-1} instead of being an LU solve, for instance, is a >>>>> Krylov solve. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Many thanks, >>>>>> >>>>>> Margarita >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jun 19 13:16:45 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 19 Jun 2012 12:16:45 -0600 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: On Tue, Jun 19, 2012 at 12:13 PM, Margarita Satraki < margarita.satraki at gmail.com> wrote: > For my problem (incompressible nonlinear elasticity) pcksp seemed to work > ok with gmres and even better with fgmres. > The best option was to use LU as PC but this requires a lot of memory. > You can just use -ksp_type preonly -pc_type lu to get a direct solution. Its not clear why you would want PCKSP. > Why do you think that would not be a good option? > Incompressible elasticity has a constraint, so it is a saddle point system. We would recommend using PC_FIELDSPLIT, splitting into blocks for the displacement and pressure. You can do this automatically using -pc_fieldsplit_detect_saddle_point. We would also proba bly recommend using AMG (ML, or GAMG, etc.) for the elastic block. You have to provide the near null modes (6 rot+trans), but there are examples of this from Mark Adams and me. Matt > Margarita > > > On 19 June 2012 19:00, Matthew Knepley wrote: > >> On Tue, Jun 19, 2012 at 11:54 AM, Margarita Satraki < >> margarita.satraki at gmail.com> wrote: >> >>> Thank you very much Matt. >>> >> >> I will just point out that this is unlikely to ever be what you want. Do >> you have a particular solver >> configuration in mind? >> >> Matt >> >> >>> Margarita >>> >>> On 19 June 2012 18:44, Matthew Knepley wrote: >>> >>>> On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < >>>> margarita.satraki at gmail.com> wrote: >>>> >>>>> If I understand correctly: >>>>> >>>>> For a system: >>>>> M^{-1} A x = M^{-1} b >>>>> we don't need to multiply M^{-1} A explicitly, but we solve M w = v >>>>> whenever needed. >>>>> >>>>> So the Krylov method is used in order to solve that system, or >>>>> equivalently to compute the vector M^{-1} v? >>>>> >>>> >>>> Yes >>>> >>>> Matt >>>> >>>> >>>>> >>>>> On 19 June 2012 17:54, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >>>>>> margarita.satraki at gmail.com> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I have difficulty understanding how PCKSP works. >>>>>>> >>>>>>> From: >>>>>>> >>>>>>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>>>>>> I understand that instead of using preconditioners, it uses Krylov >>>>>>> methods for the ''inner solvers''. >>>>>>> >>>>>>> What are the ''inner solvers''? Is there some kind of a subsystem >>>>>>> that is solved instead of applying a preconditioner? >>>>>>> >>>>>> >>>>>> Nope, its jsut like a PC: >>>>>> >>>>>> M^{-1} A x = M^{-1} b >>>>>> >>>>>> where now M^{-1} instead of being an LU solve, for instance, is a >>>>>> Krylov solve. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Many thanks, >>>>>>> >>>>>>> Margarita >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Jun 19 13:22:39 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 19 Jun 2012 13:22:39 -0500 Subject: [petsc-users] make test failed In-Reply-To: References: Message-ID: <291C0B8C-BDD1-4B3F-A4D7-18B00D906D69@mcs.anl.gov> Please run /bin/sh --version and send the output On Jun 18, 2012, at 5:21 PM, Yifei Li wrote: > Hi all, > > I got the following error when running 'make test': > > Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and PETSC_ARCH=arch-mswin-c-debug > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near unexpected token 'elif' > 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" ]; then > > > Can someone help me with this? Thanks > > Yifei From balay at mcs.anl.gov Tue Jun 19 13:27:16 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 19 Jun 2012 13:27:16 -0500 (CDT) Subject: [petsc-users] make test failed In-Reply-To: <291C0B8C-BDD1-4B3F-A4D7-18B00D906D69@mcs.anl.gov> References: <291C0B8C-BDD1-4B3F-A4D7-18B00D906D69@mcs.anl.gov> Message-ID: Ok - I think the sources were extracted with a windows tool - and not cygwin 'tar' - so the scripts [which got extracted in dos format] are misbehaving. >>>>>>> sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials $ unix2dos.exe /home/sbalay/petsc33/bin/mpiexec.uni unix2dos: converting file /home/sbalay/petsc33/bin/mpiexec.uni to DOS format ... sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials $ /home/sbalay/petsc33/bin/mpiexec.uni -n 1 ./ex2 /home/sbalay/petsc33/bin/mpiexec.uni: line 4: syntax error near unexpected token `elif' 'home/sbalay/petsc33/bin/mpiexec.uni: line 4: `elif [ $2 = "1" ]; then sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials $ dos2unix.exe /home/sbalay/petsc33/bin/mpiexec.uni dos2unix: converting file /home/sbalay/petsc33/bin/mpiexec.uni to Unix format ... sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials $ /home/sbalay/petsc33/bin/mpiexec.uni -n 1 ./ex2 Norm of error 0.000156044 iterations 6 <<<<<<< Satish On Tue, 19 Jun 2012, Barry Smith wrote: > > Please run > > /bin/sh --version > > and send the output > > > On Jun 18, 2012, at 5:21 PM, Yifei Li wrote: > > > Hi all, > > > > I got the following error when running 'make test': > > > > Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and PETSC_ARCH=arch-mswin-c-debug > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near unexpected token 'elif' > > 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" ]; then > > > > > > Can someone help me with this? Thanks > > > > Yifei > > From margarita.satraki at gmail.com Tue Jun 19 13:38:44 2012 From: margarita.satraki at gmail.com (Margarita Satraki) Date: Tue, 19 Jun 2012 19:38:44 +0100 Subject: [petsc-users] PCKSP In-Reply-To: References: Message-ID: I'll give it a go. Thanks! Margarita On 19 June 2012 19:16, Matthew Knepley wrote: > On Tue, Jun 19, 2012 at 12:13 PM, Margarita Satraki < > margarita.satraki at gmail.com> wrote: > >> For my problem (incompressible nonlinear elasticity) pcksp seemed to work >> ok with gmres and even better with fgmres. >> The best option was to use LU as PC but this requires a lot of memory. >> > > You can just use > > -ksp_type preonly > -pc_type lu > > to get a direct solution. Its not clear why you would want PCKSP. > > >> Why do you think that would not be a good option? >> > > Incompressible elasticity has a constraint, so it is a saddle point > system. We would recommend using > PC_FIELDSPLIT, splitting into blocks for the displacement and pressure. > You can do this automatically > using -pc_fieldsplit_detect_saddle_point. > > We would also proba bly recommend using AMG (ML, or GAMG, etc.) for the > elastic block. You have to provide > the near null modes (6 rot+trans), but there are examples of this from > Mark Adams and me. > > Matt > > >> Margarita >> >> >> On 19 June 2012 19:00, Matthew Knepley wrote: >> >>> On Tue, Jun 19, 2012 at 11:54 AM, Margarita Satraki < >>> margarita.satraki at gmail.com> wrote: >>> >>>> Thank you very much Matt. >>>> >>> >>> I will just point out that this is unlikely to ever be what you want. Do >>> you have a particular solver >>> configuration in mind? >>> >>> Matt >>> >>> >>>> Margarita >>>> >>>> On 19 June 2012 18:44, Matthew Knepley wrote: >>>> >>>>> On Tue, Jun 19, 2012 at 11:41 AM, Margarita Satraki < >>>>> margarita.satraki at gmail.com> wrote: >>>>> >>>>>> If I understand correctly: >>>>>> >>>>>> For a system: >>>>>> M^{-1} A x = M^{-1} b >>>>>> we don't need to multiply M^{-1} A explicitly, but we solve M w = v >>>>>> whenever needed. >>>>>> >>>>>> So the Krylov method is used in order to solve that system, or >>>>>> equivalently to compute the vector M^{-1} v? >>>>>> >>>>> >>>>> Yes >>>>> >>>>> Matt >>>>> >>>>> >>>>>> >>>>>> On 19 June 2012 17:54, Matthew Knepley wrote: >>>>>> >>>>>>> On Tue, Jun 19, 2012 at 10:33 AM, Margarita Satraki < >>>>>>> margarita.satraki at gmail.com> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I have difficulty understanding how PCKSP works. >>>>>>>> >>>>>>>> From: >>>>>>>> >>>>>>>> http://www.mcs.anl.gov/petsc/documentation/tutorials/Columbia04/DDandMultigrid.pdf >>>>>>>> I understand that instead of using preconditioners, it uses Krylov >>>>>>>> methods for the ''inner solvers''. >>>>>>>> >>>>>>>> What are the ''inner solvers''? Is there some kind of a subsystem >>>>>>>> that is solved instead of applying a preconditioner? >>>>>>>> >>>>>>> >>>>>>> Nope, its jsut like a PC: >>>>>>> >>>>>>> M^{-1} A x = M^{-1} b >>>>>>> >>>>>>> where now M^{-1} instead of being an LU solve, for instance, is a >>>>>>> Krylov solve. >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Many thanks, >>>>>>>> >>>>>>>> Margarita >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Tue Jun 19 13:48:13 2012 From: yifli82 at gmail.com (Yifei Li) Date: Tue, 19 Jun 2012 14:48:13 -0400 Subject: [petsc-users] make test failed In-Reply-To: References: <291C0B8C-BDD1-4B3F-A4D7-18B00D906D69@mcs.anl.gov> Message-ID: Thanks! Problem fixed. -Yifei On Tue, Jun 19, 2012 at 2:27 PM, Satish Balay wrote: > Ok - I think the sources were extracted with a windows tool - and not > cygwin 'tar' - so the scripts [which got extracted in dos format] are > misbehaving. > > >>>>>>> > sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials > $ unix2dos.exe /home/sbalay/petsc33/bin/mpiexec.uni > unix2dos: converting file /home/sbalay/petsc33/bin/mpiexec.uni to DOS > format ... > > sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials > $ /home/sbalay/petsc33/bin/mpiexec.uni -n 1 ./ex2 > /home/sbalay/petsc33/bin/mpiexec.uni: line 4: syntax error near unexpected > token `elif' > 'home/sbalay/petsc33/bin/mpiexec.uni: line 4: `elif [ $2 = "1" ]; then > > sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials > $ dos2unix.exe /home/sbalay/petsc33/bin/mpiexec.uni > dos2unix: converting file /home/sbalay/petsc33/bin/mpiexec.uni to Unix > format ... > > sbalay at ps3 ~/petsc33/src/ksp/ksp/examples/tutorials > $ /home/sbalay/petsc33/bin/mpiexec.uni -n 1 ./ex2 > Norm of error 0.000156044 iterations 6 > > <<<<<<< > > Satish > > > On Tue, 19 Jun 2012, Barry Smith wrote: > > > > > Please run > > > > /bin/sh --version > > > > and send the output > > > > > > On Jun 18, 2012, at 5:21 PM, Yifei Li wrote: > > > > > Hi all, > > > > > > I got the following error when running 'make test': > > > > > > Using PETSC_DIR=/home/liyi0000/petsc-3.3-p1 and > PETSC_ARCH=arch-mswin-c-debug > > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 > MPI process > > > > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > /home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: syntax error near > unexpected token 'elif' > > > 'home/liyi0000/petsc-3.3-p1/bin/mpiexec.uni: line 4: 'elif [ $2 = "1" > ]; then > > > > > > > > > Can someone help me with this? Thanks > > > > > > Yifei > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yifli82 at gmail.com Tue Jun 19 15:03:26 2012 From: yifli82 at gmail.com (Yifei Li) Date: Tue, 19 Jun 2012 16:03:26 -0400 Subject: [petsc-users] a SLEPc installation problem Message-ID: Hi, I tried to install SLEPc after successfully installing PETSc, however, I got the errors like the following when I run 'make'. P.S. I use Cygwin and Visual Studio 2008 on Windows XP c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(33) : error C2054: expec ted '(' to follow 'PETSC_EXTERN_CXX_BEGIN' c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(33) : error C2085: 'VecR egister_Comp' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(34) : error C2085: 'VecC reateComp' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(35) : error C2085: 'VecC reateCompWithVecs' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(36) : error C2085: 'VecC ompGetSubVecs' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(37) : error C2085: 'VecC ompSetSubVecs' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(40) : error C2085: 'Slep cVecSetTemplate' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(41) : error C2085: 'Slep cMatGetVecsTemplate' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(44) : error C2085: 'Slep cUpdateVectors' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(45) : error C2085: 'Slep cUpdateStrideVectors' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(46) : error C2085: 'Slep cVecMAXPBY' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(49) : error C2085: 'Slep cVecSetRandom' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(50) : error C2085: 'Slep cVecNormalize' : not in formal parameter list c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(52) : error C2061: synta x error : identifier 'PETSC_EXTERN_CXX_END' C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(102) : error C2054: expe cted '(' to follow 'PETSC_EXTERN_CXX_BEGIN' C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(102) : error C2085: 'Sle pcInitialize' : not in formal parameter list C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(103) : error C2085: 'Sle pcFinalize' : not in formal parameter list C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(104) : error C2085: 'Sle pcInitializeFortran' : not in formal parameter list C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(105) : error C2085: 'Sle pcInitialized' : not in formal parameter list C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(124) : error C2085: 'Sle pcAbs' : not in formal parameter list Yifei -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jun 19 15:16:25 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 19 Jun 2012 15:16:25 -0500 (CDT) Subject: [petsc-users] a SLEPc installation problem In-Reply-To: References: Message-ID: you'll need slepc-dev to go with petsc-3.3 Satish On Tue, 19 Jun 2012, Yifei Li wrote: > Hi, > > I tried to install SLEPc after successfully installing PETSc, however, I > got the errors like the following when I run 'make'. > > P.S. I use Cygwin and Visual Studio 2008 on Windows XP > > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(33) : error C2054: > expec > ted '(' to follow 'PETSC_EXTERN_CXX_BEGIN' > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(33) : error C2085: > 'VecR > egister_Comp' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(34) : error C2085: > 'VecC > reateComp' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(35) : error C2085: > 'VecC > reateCompWithVecs' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(36) : error C2085: > 'VecC > ompGetSubVecs' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(37) : error C2085: > 'VecC > ompSetSubVecs' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(40) : error C2085: > 'Slep > cVecSetTemplate' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(41) : error C2085: > 'Slep > cMatGetVecsTemplate' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(44) : error C2085: > 'Slep > cUpdateVectors' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(45) : error C2085: > 'Slep > cUpdateStrideVectors' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(46) : error C2085: > 'Slep > cVecMAXPBY' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(49) : error C2085: > 'Slep > cVecSetRandom' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(50) : error C2085: > 'Slep > cVecNormalize' : not in formal parameter list > c:\cygwin\home\liyi0000\slepc-3.2-p5\include\slepcvec.h(52) : error C2061: > synta > x error : identifier 'PETSC_EXTERN_CXX_END' > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(102) : error C2054: > expe > cted '(' to follow 'PETSC_EXTERN_CXX_BEGIN' > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(102) : error C2085: > 'Sle > pcInitialize' : not in formal parameter list > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(103) : error C2085: > 'Sle > pcFinalize' : not in formal parameter list > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(104) : error C2085: > 'Sle > pcInitializeFortran' : not in formal parameter list > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(105) : error C2085: > 'Sle > pcInitialized' : not in formal parameter list > C:\cygwin\home\liyi0000\SLEPC-~1.2-P\include\slepcsys.h(124) : error C2085: > 'Sle > pcAbs' : not in formal parameter list > > Yifei > From juhaj at iki.fi Wed Jun 20 03:52:55 2012 From: juhaj at iki.fi (Juha =?iso-8859-1?q?J=E4ykk=E4?=) Date: Wed, 20 Jun 2012 10:52:55 +0200 Subject: [petsc-users] strange linking problem Message-ID: <201206201053.01183.juhaj@iki.fi> Dear list, I have a strange linking problem with libpetsc.so. After running 'make PETSC_DIR=$(pwd) PETSC_ARCH=linux-gnu-cxx-opt all', the library looks like this: ~> nm linux-gnu-cxx-opt/lib/libpetsc.so|grep SAMappingInitializePackage 000000000048b4fe T SAMappingInitializePackage U _Z26SAMappingInitializePackagePKc 00000000006060b0 r _ZZ26SAMappingInitializePackageE8__func__ And, of course, make install will not fix it and therefore make test fails and all programs using it fail. Any ideas what could be causing this? It looks to me as if part of the code was compiled with mpicxx and got the names mangled in C++ fashion (hence _Z26SAMappingInitializePackagePKc) but for other parts, mpicc was used (hence no mangling and SAMappingInitializePackage). But why? I have --CC=mpicc.openmpi-gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 on the congfigure line, so the compilers should be fine. The complete configure line is: ./configure --prefix=${PREFIX}/petsc --with-shared-libraries --with- debugging=0 --with-fortran=0 --with-clanguage=C++ --with-dynamic-loading -- with-c-support --useThreads=0 --with-mpi-shared=1 --with-hdf5=1 --with-gnu- compilers=1 --with-vendor-compilers=\[pathscale,intel\] --with-hdf5- dir=${HDF5_DIR} --with-petsc-arch=linux-gnu-cxx-opt --CC=mpicc.openmpi- gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 and is the exact same line as I have used on numerous other machines, too. Any ideas what's wrong or how to debug this? Cheers, Juha -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 230 bytes Desc: This is a digitally signed message part. URL: From aron.ahmadia at kaust.edu.sa Wed Jun 20 03:55:12 2012 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 20 Jun 2012 11:55:12 +0300 Subject: [petsc-users] strange linking problem In-Reply-To: <201206201053.01183.juhaj@iki.fi> References: <201206201053.01183.juhaj@iki.fi> Message-ID: Can you send the complete configure.log to "PETSc Maint" < petsc-maint at mcs.anl.gov>? The developers are probably still sleeping right now (they're all mostly on US Central time), it's not a problem I've seen before. A On Wed, Jun 20, 2012 at 11:52 AM, Juha J?ykk? wrote: > Dear list, > > I have a strange linking problem with libpetsc.so. After running 'make > PETSC_DIR=$(pwd) PETSC_ARCH=linux-gnu-cxx-opt all', the library looks like > this: > > ~> nm linux-gnu-cxx-opt/lib/libpetsc.so|grep SAMappingInitializePackage > 000000000048b4fe T SAMappingInitializePackage > U _Z26SAMappingInitializePackagePKc > 00000000006060b0 r _ZZ26SAMappingInitializePackageE8__func__ > > And, of course, make install will not fix it and therefore make test fails > and > all programs using it fail. > > Any ideas what could be causing this? It looks to me as if part of the code > was compiled with mpicxx and got the names mangled in C++ fashion (hence > _Z26SAMappingInitializePackagePKc) but for other parts, mpicc was used > (hence > no mangling and SAMappingInitializePackage). But why? > > I have --CC=mpicc.openmpi-gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 on the > congfigure line, so the compilers should be fine. The complete configure > line > is: > > ./configure --prefix=${PREFIX}/petsc --with-shared-libraries --with- > debugging=0 --with-fortran=0 --with-clanguage=C++ --with-dynamic-loading -- > with-c-support --useThreads=0 --with-mpi-shared=1 --with-hdf5=1 --with-gnu- > compilers=1 --with-vendor-compilers=\[pathscale,intel\] --with-hdf5- > dir=${HDF5_DIR} --with-petsc-arch=linux-gnu-cxx-opt --CC=mpicc.openmpi- > gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 > > and is the exact same line as I have used on numerous other machines, too. > > Any ideas what's wrong or how to debug this? > > Cheers, > Juha > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Wed Jun 20 04:05:04 2012 From: popov at uni-mainz.de (Anton Popov) Date: Wed, 20 Jun 2012 11:05:04 +0200 Subject: [petsc-users] MatSetOption bug? In-Reply-To: References: <4FE0A39C.8030808@uni-mainz.de> Message-ID: <4FE19240.5060708@uni-mainz.de> Thanks Matt, it's clear now On 6/19/12 6:49 PM, Matthew Knepley wrote: > On Tue, Jun 19, 2012 at 10:06 AM, Anton Popov > wrote: > > Hi petsc team! > I've recently upgraded to 3.3, and currently experience problems. > One of them follows. > If I compile this simple code: > > > In 3.3, we now require that MatSetUp() is called before messing with > the storage scheme. > We should have been more specific in Changes. Moreover, this should be > a better error > message. Will fix, and see below. > > #include "petsc.h" > #undef __FUNCT__ > #define __FUNCT__ "main" > int main( int argc,char **argv ) > { > Mat A; > PetscErrorCode ierr; > PetscInt n_rows_Local=100; > PetscInt n_cols_Local=100; > ierr = PetscInitialize(&argc, &argv, PETSC_NULL, PETSC_NULL); > CHKERRQ(ierr); > ierr = MatCreate(PETSC_COMM_WORLD, &A); CHKERRQ(ierr); > ierr = MatSetSizes(A, n_rows_Local, n_cols_Local, > PETSC_DETERMINE, PETSC_DETERMINE); CHKERRQ(ierr); > ierr = MatSetType(A, MATAIJ); CHKERRQ(ierr); > > > ierr = MatSetUp(A);CHKERRQ(ierr); > > Matt > > ierr = MatSetOption(A, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); > CHKERRQ(ierr); > ierr = PetscFinalize(); CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > and run it in parallel (mpiexec -np 2 ./example) it gives me > errors like this: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Null argument, when expecting valid pointer! > [0]PETSC ERROR: Null Object: Parameter # 1! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > Commenting the line with "MatSetOption" removes the error, but > doesn't solve the problem, because I actually need to > KEEP_NONZERO_PATTERN > > Running the code sequentially, also vanishes the error. > > Please check whether it's a bug in petsc-3.3, or I should set this > option in a different way. > > Thanks, > > Anton Popov > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhaj at iki.fi Wed Jun 20 04:44:19 2012 From: juhaj at iki.fi (Juha =?iso-8859-15?q?J=E4ykk=E4?=) Date: Wed, 20 Jun 2012 11:44:19 +0200 Subject: [petsc-users] strange linking problem In-Reply-To: References: <201206201053.01183.juhaj@iki.fi> Message-ID: <201206201144.19679.juhaj@iki.fi> > Can you send the complete configure.log to "PETSc Maint" < I was looking at the log myself, but as it is over 47000 lines long and I have no idea what I am looking for, it is hopeless. There is one thing that I notice, though: Check c-support c++-support and other misc tests Defined "USE_EXTERN_CXX" to "1" Turning off C++ name mangling C language is Cxx Defined "CLANGUAGE_CXX" to "1" Now, C++ name mangling is *certainly* on, since I get symbols like _Z26SAMappingInitializePackagePKc which are not present in the source code (source only has the unmangled SAMappingInitializePackage). Cheers, Juha -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 230 bytes Desc: This is a digitally signed message part. URL: From matijakecman at gmail.com Wed Jun 20 04:56:27 2012 From: matijakecman at gmail.com (Matija Kecman) Date: Wed, 20 Jun 2012 09:56:27 +0000 (UTC) Subject: [petsc-users] Invitation to connect on LinkedIn Message-ID: <537704395.19266144.1340186187182.JavaMail.app@ela4-bed82.prod> LinkedIn ------------ PETSc, I'd like to add you to my professional network on LinkedIn. - Matija Matija Kecman Student at University of Cambridge Cambridge, United Kingdom Confirm that you know Matija Kecman: https://www.linkedin.com/e/-r9oj6w-h3o8948b-6j/isd/7563030205/1ktWUteT/?hs=false&tok=2Vn1PuFFOQQRg1 -- You are receiving Invitation to Connect emails. Click to unsubscribe: http://www.linkedin.com/e/-r9oj6w-h3o8948b-6j/NPBLyes6_CJvfaFX95qTY0Fn_yVIxe9EWtXp/goo/petsc-users%40mcs%2Eanl%2Egov/20061/I2563354815_1/?hs=false&tok=0zScGjtQeQQRg1 (c) 2012 LinkedIn Corporation. 2029 Stierlin Ct, Mountain View, CA 94043, USA. -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Wed Jun 20 10:00:32 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Wed, 20 Jun 2012 15:00:32 +0000 Subject: [petsc-users] replacing Schur complement in pcfieldsplit Message-ID: In order to implement SIMPLE-type preconditioners for the incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) using the PCFieldSplit framework, it looks like I need to replace inv(A00) by some cheap approximation 1) in the Schur complement 2) in the L and/or U of the LDU factorization 3) while keeping A00 in the D What would be the best way? dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From knepley at gmail.com Wed Jun 20 10:04:12 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 20 Jun 2012 09:04:12 -0600 Subject: [petsc-users] replacing Schur complement in pcfieldsplit In-Reply-To: References: Message-ID: On Wed, Jun 20, 2012 at 9:00 AM, Klaij, Christiaan wrote: > > In order to implement SIMPLE-type preconditioners for the > incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) > using the PCFieldSplit framework, it looks like I need to replace > inv(A00) by some cheap approximation > > 1) in the Schur complement > When you have a Schur FS, the '0' solver is this approximation. > 2) in the L and/or U of the LDU factorization > You can use whatever PC you want for the solver mentioned above. > 3) while keeping A00 in the D > I think what you want here is -pc_fieldsplit_real_diagonal. I would be really nice to make a PETSc example that did SIMPLE. I was going to do this, but if you do it first, I will put it in. Matt > What would be the best way? > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Jun 20 10:43:38 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 20 Jun 2012 17:43:38 +0200 Subject: [petsc-users] Ambiguity of KSPCGNE Message-ID: <4FE1EFAA.3020608@gfz-potsdam.de> Hello, I'm a bit confused about the KSPCGNE. First of all, is CGLS^1 and implemented CGNE are the same (or I mix it up with CGNR)? I don't know what notation is more classical, but CGLS seems to be more common. It is claimed: */Applies the preconditioned conjugate gradient method to the normal equations without explicitly forming A^t*A /* Does that mean I have to provide A to KSP? In this case the application of the method is quite restricted since all practical least squares problems formulated in form of normal equations are solved with regularization, e.g.: (A'A + \lamba I)x = A'b Assume I have A computed and use matrix free approach to represent (A'A + \lamba I) without ever forming it, so what should I do then to apply KSPCGNE? Thanks. 1. Bjorck, A., 1996. Numerical Methods for Least Squares Problems, p. 288 -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 20 16:27:09 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 20 Jun 2012 16:27:09 -0500 Subject: [petsc-users] strange linking problem In-Reply-To: <201206201053.01183.juhaj@iki.fi> References: <201206201053.01183.juhaj@iki.fi> Message-ID: <8AF7ECE7-0FE7-4E36-A392-7C263406FF54@mcs.anl.gov> I cannot find SAMappingInitializePackage in PETSc. This option > -with-c-support means that PETSc is compiled with the C++ compiler but without C++ name mangling turned on (so that functions can be called from code compiled with the C compiler). There is rarely a need to use this option since if you want to use PETSc from C just build it with the C compiler. Why are you using this option? Now your problem comes up because whoever provided SAMappingInitializePackage did not do it correctly. Likely the prototype for it does not wrap it in extern C so that when called it gets mangled but when the code is compiled it does not mangle. Please switch to petsc-3.3 where fortunately all that SAMappingInitializePackage has been removed and the problem will go away. Barry The person who put this incorrect SAMappingInitializePackage no longer has write access to PETSc source code so cannot cause these problems again in the future. On Jun 20, 2012, at 3:52 AM, Juha J?ykk? wrote: > Dear list, > > I have a strange linking problem with libpetsc.so. After running 'make > PETSC_DIR=$(pwd) PETSC_ARCH=linux-gnu-cxx-opt all', the library looks like > this: > > ~> nm linux-gnu-cxx-opt/lib/libpetsc.so|grep SAMappingInitializePackage > 000000000048b4fe T SAMappingInitializePackage > U _Z26SAMappingInitializePackagePKc > 00000000006060b0 r _ZZ26SAMappingInitializePackageE8__func__ > > And, of course, make install will not fix it and therefore make test fails and > all programs using it fail. > > Any ideas what could be causing this? It looks to me as if part of the code > was compiled with mpicxx and got the names mangled in C++ fashion (hence > _Z26SAMappingInitializePackagePKc) but for other parts, mpicc was used (hence > no mangling and SAMappingInitializePackage). But why? > > I have --CC=mpicc.openmpi-gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 on the > congfigure line, so the compilers should be fine. The complete configure line > is: > > ./configure --prefix=${PREFIX}/petsc --with-shared-libraries --with- > debugging=0 --with-fortran=0 --with-clanguage=C++ --with-dynamic-loading -- > with-c-support --useThreads=0 --with-mpi-shared=1 --with-hdf5=1 --with-gnu- > compilers=1 --with-vendor-compilers=\[pathscale,intel\] --with-hdf5- > dir=${HDF5_DIR} --with-petsc-arch=linux-gnu-cxx-opt --CC=mpicc.openmpi- > gcc-1.4.4 --CXX=mpicxx.openmpi-gcc-1.4.4 > > and is the exact same line as I have used on numerous other machines, too. > > Any ideas what's wrong or how to debug this? > > Cheers, > Juha From C.Klaij at marin.nl Thu Jun 21 02:17:49 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Thu, 21 Jun 2012 07:17:49 +0000 Subject: [petsc-users] replacing Schur complement in pcfieldsplit Message-ID: > > > > In order to implement SIMPLE-type preconditioners for the > > incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) > > using the PCFieldSplit framework, it looks like I need to replace > > inv(A00) by some cheap approximation > > > > 1) in the Schur complement > > > > When you have a Schur FS, the '0' solver is this approximation. > > > > 2) in the L and/or U of the LDU factorization > > > > You can use whatever PC you want for the solver mentioned above. > > > > 3) while keeping A00 in the D > > > > I think what you want here is -pc_fieldsplit_real_diagonal. Let me get this straight. Looking at the full LDU factorization of the block matrix. Citing from the manual: For the Schur complement preconditioner if J = ( A00 A01 ) ( A10 A11 ) the preconditioner using full factorization is ( I -A10 ksp(A00) ) ( inv(A00) 0 ) ( I 0 ) ( 0 I ) ( 0 ksp(S) ) ( -A10 ksp(A00) I ) Clearly inv(A00) occurs four times, right? In L and in U and twice in D. Now if I somehow overrule the '0' solver with my approximation and use -pc_fieldsplit_real_diagonal, the effect would be that inv(A00) is replaced in L, in U and in S but not in the 00-block of D? And what's the function name corresponding to -pc_fieldsplit_real_diagonal? > > I would be really nice to make a PETSc example that did SIMPLE. I was > going to do this, but if you do it first, I will put it in. Working on it. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From goxberry at gmail.com Thu Jun 21 07:59:22 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Thu, 21 Jun 2012 08:59:22 -0400 Subject: [petsc-users] Using -ts_type sundials with -snes-fd Message-ID: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> Running the following example from PETSC 3.3.0-dev (changeset: 23631:0e86ac5e4170) /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober -snes_fd -ts_type sundials gives the following output steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, linits 3739 WARNING! There are options you set that were not used! WARNING! could be spelling mistake, etc! Option left: name:-snes_fd no value Just to confirm, is it currently impossible to use a finite difference Jacobian matrix in concert with CVODE? If so, could this feature be implemented in a future release? I currently rely on Sundials to integrate stiff systems of ODEs, and for my application, it is impractical to derive an analytical Jacobian matrix. (It is an issue I've discussed both with Jed and Matt on another forum.) Cheers, Geoff From knepley at gmail.com Thu Jun 21 08:14:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Jun 2012 07:14:47 -0600 Subject: [petsc-users] replacing Schur complement in pcfieldsplit In-Reply-To: References: Message-ID: On Thu, Jun 21, 2012 at 1:17 AM, Klaij, Christiaan wrote: > > > > > > In order to implement SIMPLE-type preconditioners for the > > > incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) > > > using the PCFieldSplit framework, it looks like I need to replace > > > inv(A00) by some cheap approximation > > > > > > 1) in the Schur complement > > > > > > > When you have a Schur FS, the '0' solver is this approximation. > > > > > > > 2) in the L and/or U of the LDU factorization > > > > > > > You can use whatever PC you want for the solver mentioned above. > > > > > > > 3) while keeping A00 in the D > > > > > > > I think what you want here is -pc_fieldsplit_real_diagonal. > > Let me get this straight. Looking at the full LDU factorization > of the block matrix. Citing from the manual: > > For the Schur complement preconditioner if > J = ( A00 A01 ) > ( A10 A11 ) > > the preconditioner using full factorization is > ( I -A10 ksp(A00) ) ( inv(A00) 0 ) ( I 0 ) > ( 0 I ) ( 0 ksp(S) ) ( -A10 ksp(A00) I ) > Yes. > Clearly inv(A00) occurs four times, right? In L and in U and > twice in D. Now if I somehow overrule the '0' solver with my > Yes > approximation and use -pc_fieldsplit_real_diagonal, the effect > would be that inv(A00) is replaced in L, in U and in S but not in > the 00-block of D? > No. What this says is that we should use the action of the actual matrix rather than the preconditioner matrix in the solver. I now think I have a better idea what you want, but it would be helpful if you wrote it out completely in linear algebra notation, as above. Right now, we use the same KSP for all 4 places above. Using different KSPs would require a small code change, which I can make if you give me a better idea what you want. > And what's the function name corresponding to > -pc_fieldsplit_real_diagonal? > We have not put one in yet. Thanks, Matt > > > > I would be really nice to make a PETSc example that did SIMPLE. I was > > going to do this, but if you do it first, I will put it in. > > Working on it. > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Thu Jun 21 08:53:18 2012 From: prbrune at gmail.com (Peter Brune) Date: Thu, 21 Jun 2012 08:53:18 -0500 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> Message-ID: Note that in the code in ts/impls/implicit/sundials it says: This uses its own nonlinear solver and krylov method so PETSc SNES and KSP options do not apply... - Peter On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: > Running the following example from PETSC 3.3.0-dev (changeset: > 23631:0e86ac5e4170) > > /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober > -snes_fd -ts_type sundials > > gives the following output > > steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, > linits 3739 > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-snes_fd no value > > Just to confirm, is it currently impossible to use a finite difference > Jacobian matrix in concert with CVODE? If so, could this feature be > implemented in a future release? I currently rely on Sundials to integrate > stiff systems of ODEs, and for my application, it is impractical to derive > an analytical Jacobian matrix. (It is an issue I've discussed both with Jed > and Matt on another forum.) > > Cheers, > > Geoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From goxberry at gmail.com Thu Jun 21 09:40:32 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Thu, 21 Jun 2012 10:40:32 -0400 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> Message-ID: <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> Peter, Just wanted to make sure there wasn't some Sundials-specific option for finite difference Jacobians that I was missing; despite reading the manual, it's a large package, and it's easy to miss things. If that's the case, I'd like to make a feature request for such an option. Geoff On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: > Note that in the code in ts/impls/implicit/sundials it says: > > This uses its own nonlinear solver and krylov method so PETSc SNES and KSP options do not apply... > > - Peter > > On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: > Running the following example from PETSC 3.3.0-dev (changeset: 23631:0e86ac5e4170) > > /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober -snes_fd -ts_type sundials > > gives the following output > > steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, linits 3739 > WARNING! There are options you set that were not used! > WARNING! could be spelling mistake, etc! > Option left: name:-snes_fd no value > > Just to confirm, is it currently impossible to use a finite difference Jacobian matrix in concert with CVODE? If so, could this feature be implemented in a future release? I currently rely on Sundials to integrate stiff systems of ODEs, and for my application, it is impractical to derive an analytical Jacobian matrix. (It is an issue I've discussed both with Jed and Matt on another forum.) > > Cheers, > > Geoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Thu Jun 21 09:45:10 2012 From: prbrune at gmail.com (Peter Brune) Date: Thu, 21 Jun 2012 09:45:10 -0500 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> Message-ID: What do you see when you run with -ts_view? - Peter On Thu, Jun 21, 2012 at 9:40 AM, Geoff Oxberry wrote: > Peter, > > Just wanted to make sure there wasn't some Sundials-specific option for > finite difference Jacobians that I was missing; despite reading the manual, > it's a large package, and it's easy to miss things. If that's the case, I'd > like to make a feature request for such an option. > > Geoff > > On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: > > Note that in the code in ts/impls/implicit/sundials it says: > > This uses its own nonlinear solver and krylov method so PETSc SNES and KSP > options do not apply... > > - Peter > On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: > >> Running the following example from PETSC 3.3.0-dev (changeset: >> 23631:0e86ac5e4170) >> >> /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober >> -snes_fd -ts_type sundials >> >> gives the following output >> >> steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, >> linits 3739 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-snes_fd no value >> >> Just to confirm, is it currently impossible to use a finite difference >> Jacobian matrix in concert with CVODE? If so, could this feature be >> implemented in a future release? I currently rely on Sundials to integrate >> stiff systems of ODEs, and for my application, it is impractical to derive >> an analytical Jacobian matrix. (It is an issue I've discussed both with Jed >> and Matt on another forum.) >> >> Cheers, >> >> Geoff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 21 09:48:22 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Jun 2012 08:48:22 -0600 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> Message-ID: On Thu, Jun 21, 2012 at 8:45 AM, Peter Brune wrote: > What do you see when you run with -ts_view? > > - Peter > > > On Thu, Jun 21, 2012 at 9:40 AM, Geoff Oxberry wrote: > >> Peter, >> >> Just wanted to make sure there wasn't some Sundials-specific option for >> finite difference Jacobians that I was missing; despite reading the manual, >> it's a large package, and it's easy to miss things. If that's the case, I'd >> like to make a feature request for such an option. >> > If I understand correctly, you want a MF Jacobian with Sundials. We can't do that because Sundials is completely closed package, which we cannot pry apart to insert something like this. The alternative is to use the stuff solvers we currently have in TS. I thought that you had used the Rosenbrock-W stuff. Is this sufficient? Thanks, Matt > Geoff >> >> On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: >> >> Note that in the code in ts/impls/implicit/sundials it says: >> >> This uses its own nonlinear solver and krylov method so PETSc SNES and >> KSP options do not apply... >> >> - Peter >> On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: >> >>> Running the following example from PETSC 3.3.0-dev (changeset: >>> 23631:0e86ac5e4170) >>> >>> /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober >>> -snes_fd -ts_type sundials >>> >>> gives the following output >>> >>> steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, >>> linits 3739 >>> WARNING! There are options you set that were not used! >>> WARNING! could be spelling mistake, etc! >>> Option left: name:-snes_fd no value >>> >>> Just to confirm, is it currently impossible to use a finite difference >>> Jacobian matrix in concert with CVODE? If so, could this feature be >>> implemented in a future release? I currently rely on Sundials to integrate >>> stiff systems of ODEs, and for my application, it is impractical to derive >>> an analytical Jacobian matrix. (It is an issue I've discussed both with Jed >>> and Matt on another forum.) >>> >>> Cheers, >>> >>> Geoff >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Thu Jun 21 09:52:00 2012 From: prbrune at gmail.com (Peter Brune) Date: Thu, 21 Jun 2012 09:52:00 -0500 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> Message-ID: I asked my question because it might be happening automatically. His output looked like the equation was solved. A brief google search on this makes it look like they do this anyways. In addition: http://www.mcs.anl.gov/petsc/petsc-dev/src/ts/impls/implicit/sundials/sundials.c.html#TSView_Sundials directly refers to "Sundials no. of rhs calls for finite diff. Jacobian-vector evals." - Peter On Thu, Jun 21, 2012 at 9:48 AM, Matthew Knepley wrote: > On Thu, Jun 21, 2012 at 8:45 AM, Peter Brune wrote: > >> What do you see when you run with -ts_view? >> >> - Peter >> >> >> On Thu, Jun 21, 2012 at 9:40 AM, Geoff Oxberry wrote: >> >>> Peter, >>> >>> Just wanted to make sure there wasn't some Sundials-specific option for >>> finite difference Jacobians that I was missing; despite reading the manual, >>> it's a large package, and it's easy to miss things. If that's the case, I'd >>> like to make a feature request for such an option. >>> >> > If I understand correctly, you want a MF Jacobian with Sundials. We can't > do that because Sundials is completely > closed package, which we cannot pry apart to insert something like this. > The alternative is to use the stuff solvers > we currently have in TS. I thought that you had used the Rosenbrock-W > stuff. Is this sufficient? > > Thanks, > > Matt > > >> Geoff >>> >>> On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: >>> >>> Note that in the code in ts/impls/implicit/sundials it says: >>> >>> This uses its own nonlinear solver and krylov method so PETSc SNES and >>> KSP options do not apply... >>> >>> - Peter >>> On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: >>> >>>> Running the following example from PETSC 3.3.0-dev (changeset: >>>> 23631:0e86ac5e4170) >>>> >>>> /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober >>>> -snes_fd -ts_type sundials >>>> >>>> gives the following output >>>> >>>> steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, >>>> linits 3739 >>>> WARNING! There are options you set that were not used! >>>> WARNING! could be spelling mistake, etc! >>>> Option left: name:-snes_fd no value >>>> >>>> Just to confirm, is it currently impossible to use a finite difference >>>> Jacobian matrix in concert with CVODE? If so, could this feature be >>>> implemented in a future release? I currently rely on Sundials to integrate >>>> stiff systems of ODEs, and for my application, it is impractical to derive >>>> an analytical Jacobian matrix. (It is an issue I've discussed both with Jed >>>> and Matt on another forum.) >>>> >>>> Cheers, >>>> >>>> Geoff >>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paeanball at gmail.com Thu Jun 21 13:20:50 2012 From: paeanball at gmail.com (Bao Kai) Date: Thu, 21 Jun 2012 21:20:50 +0300 Subject: [petsc-users] How to use bicgstab2, bicgstab3 and bicgstab4 Message-ID: Hi, all, I am using petsc3.2 at the moment. In the src directory, I found the ksp bcgsl which is intended for BICGSTAB(L). My question is how to change the value of L. Should I use -ksp_bcgsl_ell to set different ell values? Best Regards, Kai From goxberry at gmail.com Thu Jun 21 14:46:50 2012 From: goxberry at gmail.com (Geoff Oxberry) Date: Thu, 21 Jun 2012 15:46:50 -0400 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> Message-ID: <1EFDEDC9-4D04-43BA-8F3E-596B1CEC201C@gmail.com> Peter, You are correct. Here is the output (now with petsc-dev changeset: 23668:a876ee8e66fd): goxberry$ ./ex8 -problem_type rober -ts_type sundials -ts_view TS Object: 1 MPI processes type: sundials maximum steps=1000 maximum time=1e+11 total number of nonlinear solver iterations=3739 total number of nonlinear solve failures=0 total number of linear solver iterations=3739 total number of rejected steps=0 Sundials integrater does not use SNES! Sundials integrater type BDF: backward differentiation formula Sundials abs tol 1e-06 rel tol 1e-06 Sundials linear solver tolerance factor 0.05 Sundials max dimension of Krylov subspace 5 Sundials using unmodified (classical) Gram-Schmidt for orthogonalization in GMRES Sundials suggested factor for tolerance scaling 1 Sundials cumulative number of internal steps 1000 Sundials no. of calls to rhs function 1931 Sundials no. of calls to linear solver setup function 0 Sundials no. of error test failures 16 Sundials no. of nonlinear solver iterations 1930 Sundials no. of nonlinear convergence failure 204 Sundials no. of linear iterations 3739 Sundials no. of linear convergence failures 0 PC Object: 1 MPI processes type: none Sundials no. of preconditioner evaluations 0 Sundials no. of preconditioner solves 0 Sundials no. of Jacobian-vector product evaluations 3739 Sundials no. of rhs calls for finite diff. Jacobian-vector evals 3739 steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, linits 3739 I use Sundials mostly with finite difference Jacobians, but I also provide Jacobian matrices where I can (which would be another helpful feature). Cheers, Geoff On Jun 21, 2012, at 10:52 AM, Peter Brune wrote: > I asked my question because it might be happening automatically. His output looked like the equation was solved. A brief google search on this makes it look like they do this anyways. In addition: > > http://www.mcs.anl.gov/petsc/petsc-dev/src/ts/impls/implicit/sundials/sundials.c.html#TSView_Sundials > > directly refers to "Sundials no. of rhs calls for finite diff. Jacobian-vector evals." > > - Peter > > On Thu, Jun 21, 2012 at 9:48 AM, Matthew Knepley wrote: > On Thu, Jun 21, 2012 at 8:45 AM, Peter Brune wrote: > What do you see when you run with -ts_view? > > - Peter > > > On Thu, Jun 21, 2012 at 9:40 AM, Geoff Oxberry wrote: > Peter, > > Just wanted to make sure there wasn't some Sundials-specific option for finite difference Jacobians that I was missing; despite reading the manual, it's a large package, and it's easy to miss things. If that's the case, I'd like to make a feature request for such an option. > > If I understand correctly, you want a MF Jacobian with Sundials. We can't do that because Sundials is completely > closed package, which we cannot pry apart to insert something like this. The alternative is to use the stuff solvers > we currently have in TS. I thought that you had used the Rosenbrock-W stuff. Is this sufficient? > > Thanks, > > Matt > > Geoff > > On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: > >> Note that in the code in ts/impls/implicit/sundials it says: >> >> This uses its own nonlinear solver and krylov method so PETSc SNES and KSP options do not apply... >> >> - Peter >> >> On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: >> Running the following example from PETSC 3.3.0-dev (changeset: 23631:0e86ac5e4170) >> >> /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober -snes_fd -ts_type sundials >> >> gives the following output >> >> steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, linits 3739 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-snes_fd no value >> >> Just to confirm, is it currently impossible to use a finite difference Jacobian matrix in concert with CVODE? If so, could this feature be implemented in a future release? I currently rely on Sundials to integrate stiff systems of ODEs, and for my application, it is impractical to derive an analytical Jacobian matrix. (It is an issue I've discussed both with Jed and Matt on another forum.) >> >> Cheers, >> >> Geoff > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jun 21 15:03:08 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 21 Jun 2012 14:03:08 -0600 Subject: [petsc-users] How to use bicgstab2, bicgstab3 and bicgstab4 In-Reply-To: References: Message-ID: On Thu, Jun 21, 2012 at 12:20 PM, Bao Kai wrote: > Hi, all, > > I am using petsc3.2 at the moment. > > In the src directory, I found the ksp bcgsl which is intended for > BICGSTAB(L). > > My question is how to change the value of L. > > Should I use -ksp_bcgsl_ell to set different ell values? > Yes. Matt > Best Regards, > Kai > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 21 17:32:48 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 21 Jun 2012 17:32:48 -0500 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: <4FE1EFAA.3020608@gfz-potsdam.de> References: <4FE1EFAA.3020608@gfz-potsdam.de> Message-ID: On Jun 20, 2012, at 10:43 AM, Alexander Grayver wrote: > Hello, > > I'm a bit confused about the KSPCGNE. First of all, is CGLS^1 and implemented CGNE are the same (or I mix it up with CGNR)? I don't know what notation is more classical, but CGLS seems to be more common. > It is claimed: It is the same as this: http://www.stanford.edu/group/SOL/software/cgls.html > > Applies the preconditioned conjugate gradient method to the normal equations without explicitly forming A^t*A > > Does that mean I have to provide A to KSP? Yes you provide A. > In this case the application of the method is quite restricted since all practical least squares problems formulated in form of normal equations are solved with regularization, e.g.: > > (A'A + \lamba I)x = A'b Yes it is restrictive. There is no concept of lambda in CGNE in PETSc > > Assume I have A computed and use matrix free approach to represent (A'A + \lamba I) without ever forming it, so what should I do then to apply KSPCGNE? If you supply a shell matrix that applies (A'A + \lamba I) why not just use KSPCG? But if you provide this shell matrix, how do you plan to apply a preconditioner? Barry > > Thanks. > > 1. Bjorck, A., 1996. Numerical Methods for Least Squares Problems, p. 288 > -- > Regards, > Alexander > From jedbrown at mcs.anl.gov Fri Jun 22 02:30:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 21 Jun 2012 23:30:09 -0800 Subject: [petsc-users] Using -ts_type sundials with -snes-fd In-Reply-To: <1EFDEDC9-4D04-43BA-8F3E-596B1CEC201C@gmail.com> References: <6542B446-8388-4C62-BB1A-E113CA097775@gmail.com> <199ACC28-984F-4B87-8E79-13B578ED664A@gmail.com> <1EFDEDC9-4D04-43BA-8F3E-596B1CEC201C@gmail.com> Message-ID: Geoff, just add -pc_type lu to the options below, it will use a (sparse) direct solver as a preconditioner. Sundials is still using matrix-free finite differencing for Jacobian-free Newton-Krylov, and we can't control how often the Jacobian is recomputed, but we can choose whatever we want as a preconditioner. The Rosenbrock methods are pretty slick in that they only need a linear solve per stage and that the Jacobian is only recomputed once per step. If you have a problem where Jacobian evaluation is very expensive, we can make methods with many stages (to further amortize setup cost; good unless it causes more step rejections). [...] PC Object: 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5, needed 1 Factored matrix follows: Matrix Object: 1 MPI processes type: seqaij rows=3, cols=3 package used to perform factorization: petsc total: nonzeros=9, allocated nonzeros=9 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=3, cols=3 total: nonzeros=9, allocated nonzeros=15 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 1 nodes, limit used is 5 Sundials no. of preconditioner evaluations 330 Sundials no. of preconditioner solves 3497 Sundials no. of Jacobian-vector product evaluations 1719 Sundials no. of rhs calls for finite diff. Jacobian-vector evals 1719 steps 1000 (0 rejected, 0 SNES fails), ftime 1.97026e+10, nonlinits 1719, linits 1719 On Thu, Jun 21, 2012 at 11:46 AM, Geoff Oxberry wrote: > Peter, > > You are correct. Here is the output (now with petsc-dev changeset: > 23668:a876ee8e66fd): > > goxberry$ ./ex8 -problem_type rober -ts_type sundials -ts_view > TS Object: 1 MPI processes > type: sundials > maximum steps=1000 > maximum time=1e+11 > total number of nonlinear solver iterations=3739 > total number of nonlinear solve failures=0 > total number of linear solver iterations=3739 > total number of rejected steps=0 > Sundials integrater does not use SNES! > Sundials integrater type BDF: backward differentiation formula > Sundials abs tol 1e-06 rel tol 1e-06 > Sundials linear solver tolerance factor 0.05 > Sundials max dimension of Krylov subspace 5 > Sundials using unmodified (classical) Gram-Schmidt for orthogonalization > in GMRES > Sundials suggested factor for tolerance scaling 1 > Sundials cumulative number of internal steps 1000 > Sundials no. of calls to rhs function 1931 > Sundials no. of calls to linear solver setup function 0 > Sundials no. of error test failures 16 > Sundials no. of nonlinear solver iterations 1930 > Sundials no. of nonlinear convergence failure 204 > Sundials no. of linear iterations 3739 > Sundials no. of linear convergence failures 0 > PC Object: 1 MPI processes > type: none > Sundials no. of preconditioner evaluations 0 > Sundials no. of preconditioner solves 0 > Sundials no. of Jacobian-vector product evaluations 3739 > Sundials no. of rhs calls for finite diff. Jacobian-vector evals 3739 > steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, > linits 3739 > > I use Sundials mostly with finite difference Jacobians, but I also provide > Jacobian matrices where I can (which would be another helpful feature). > > Cheers, > > Geoff > > On Jun 21, 2012, at 10:52 AM, Peter Brune wrote: > > I asked my question because it might be happening automatically. His > output looked like the equation was solved. A brief google search on this > makes it look like they do this anyways. In addition: > > > http://www.mcs.anl.gov/petsc/petsc-dev/src/ts/impls/implicit/sundials/sundials.c.html#TSView_Sundials > > directly refers to "Sundials no. of rhs calls for finite diff. > Jacobian-vector evals." > > - Peter > > On Thu, Jun 21, 2012 at 9:48 AM, Matthew Knepley wrote: > >> On Thu, Jun 21, 2012 at 8:45 AM, Peter Brune wrote: >> >>> What do you see when you run with -ts_view? >>> >>> - Peter >>> >>> >>> On Thu, Jun 21, 2012 at 9:40 AM, Geoff Oxberry wrote: >>> >>>> Peter, >>>> >>>> Just wanted to make sure there wasn't some Sundials-specific option for >>>> finite difference Jacobians that I was missing; despite reading the manual, >>>> it's a large package, and it's easy to miss things. If that's the case, I'd >>>> like to make a feature request for such an option. >>>> >>> >> If I understand correctly, you want a MF Jacobian with Sundials. We can't >> do that because Sundials is completely >> closed package, which we cannot pry apart to insert something like this. >> The alternative is to use the stuff solvers >> we currently have in TS. I thought that you had used the Rosenbrock-W >> stuff. Is this sufficient? >> >> Thanks, >> >> Matt >> >> >>> Geoff >>>> >>>> On Jun 21, 2012, at 9:53 AM, Peter Brune wrote: >>>> >>>> Note that in the code in ts/impls/implicit/sundials it says: >>>> >>>> This uses its own nonlinear solver and krylov method so PETSc SNES and >>>> KSP options do not apply... >>>> >>>> - Peter >>>> On Jun 21, 2012 7:59 AM, "Geoff Oxberry" wrote: >>>> >>>>> Running the following example from PETSC 3.3.0-dev (changeset: >>>>> 23631:0e86ac5e4170) >>>>> >>>>> /path/to/petsc-dev/src/ts/examples/tutorials/ex8 -problem_type rober >>>>> -snes_fd -ts_type sundials >>>>> >>>>> gives the following output >>>>> >>>>> steps 1000 (0 rejected, 0 SNES fails), ftime 744.845, nonlinits 3739, >>>>> linits 3739 >>>>> WARNING! There are options you set that were not used! >>>>> WARNING! could be spelling mistake, etc! >>>>> Option left: name:-snes_fd no value >>>>> >>>>> Just to confirm, is it currently impossible to use a finite difference >>>>> Jacobian matrix in concert with CVODE? If so, could this feature be >>>>> implemented in a future release? I currently rely on Sundials to integrate >>>>> stiff systems of ODEs, and for my application, it is impractical to derive >>>>> an analytical Jacobian matrix. (It is an issue I've discussed both with Jed >>>>> and Matt on another forum.) >>>>> >>>>> Cheers, >>>>> >>>>> Geoff >>>> >>>> >>>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From C.Klaij at marin.nl Fri Jun 22 02:46:06 2012 From: C.Klaij at marin.nl (Klaij, Christiaan) Date: Fri, 22 Jun 2012 07:46:06 +0000 Subject: [petsc-users] replacing Schur complement in pcfieldsplit Message-ID: > > > > > > > > In order to implement SIMPLE-type preconditioners for the > > > > incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) > > > > using the PCFieldSplit framework, it looks like I need to replace > > > > inv(A00) by some cheap approximation > > > > > > > > 1) in the Schur complement > > > > > > > > > > When you have a Schur FS, the '0' solver is this approximation. > > > > > > > > > > 2) in the L and/or U of the LDU factorization > > > > > > > > > > You can use whatever PC you want for the solver mentioned above. > > > > > > > > > > 3) while keeping A00 in the D > > > > > > > > > > I think what you want here is -pc_fieldsplit_real_diagonal. > > > > Let me get this straight. Looking at the full LDU factorization > > of the block matrix. Citing from the manual: > > > > For the Schur complement preconditioner if > > J = ( A00 A01 ) > > ( A10 A11 ) > > > > the preconditioner using full factorization is > > ( I -A10 ksp(A00) ) ( inv(A00) 0 ) ( I 0 ) > > ( 0 I ) ( 0 ksp(S) ) ( -A10 ksp(A00) I ) > > > > Yes. > > > > Clearly inv(A00) occurs four times, right? In L and in U and > > twice in D. Now if I somehow overrule the '0' solver with my > > > > Yes > > > > approximation and use -pc_fieldsplit_real_diagonal, the effect > > would be that inv(A00) is replaced in L, in U and in S but not in > > the 00-block of D? > > > > No. What this says is that we should use the action of the > actual matrix rather than the preconditioner matrix in the solver. Odd. Don't you *always* need the action of the actual matrix (and of the preconditioner) in a Krylov subspace method? > > I now think I have a better idea what you want, but it would be > helpful if you wrote it out completely in linear algebra notation, as > above. Right now, we use the same KSP for all 4 places above. > Using different KSPs would require a small code change, which I > can make if you give me a better idea what you want. Maybe there is a mistake in the manual, shouldn't it be ( I -A01 ksp(A00) ) ( 0 I ) in the factorization above instead of -A10 ksp(A00)? SIMPLE-type preconditioners are usually written as: ( I -A01 dA00^(-1) ) ( A00 0 )^(-1) ( 0 I ) ( A10 S ) with S = A11 - A10 dA00^(-1) A01, where dA00^(-1) is the inverse of the diagonal of A00. Therefore it only requires one solve for A00 and one solve for S. > > > > And what's the function name corresponding to > > -pc_fieldsplit_real_diagonal? > > > > We have not put one in yet. Please let me know when you do. > > Thanks, > > Matt dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:C.Klaij at marin.nl T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl From agrayver at gfz-potsdam.de Fri Jun 22 02:54:39 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Fri, 22 Jun 2012 09:54:39 +0200 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: References: <4FE1EFAA.3020608@gfz-potsdam.de> Message-ID: <4FE424BF.8080809@gfz-potsdam.de> Hi Barry, >> In this case the application of the method is quite restricted since all practical least squares problems formulated in form of normal equations are solved with regularization, e.g.: >> >> (A'A + \lamba I)x = A'b > Yes it is restrictive. There is no concept of lambda in CGNE in PETSc In this case, since there is LSQR in PETSc, there is hardly any reason to use CGNE. >> Assume I have A computed and use matrix free approach to represent (A'A + \lamba I) without ever forming it, so what should I do then to apply KSPCGNE? > If you supply a shell matrix that applies (A'A + \lamba I) why not just use KSPCG? That is what I do at the moment. However, as far as I understand, CGLS is not just about shifting original matrix with some lambda, it has other advantages over CG for normal equations. > But if you provide this shell matrix, how do you plan to apply a preconditioner? One can easily compute diag(A'A + \lamba I), thanks to MatGetColumnNorms, and thus Jacobi is possible. Since the matrix is diagonally dominant, in my case it is enough to converge. Although convergence for normal equations does not imply accurate solution, so that one needs CGLS or LSQR. -- Regards, Alexander From andreas.hauffe at tu-dresden.de Fri Jun 22 06:52:25 2012 From: andreas.hauffe at tu-dresden.de (Andreas Hauffe) Date: Fri, 22 Jun 2012 13:52:25 +0200 Subject: [petsc-users] PETSC3.2 and MUMPS 4.10.0 slower than PETSC3.1 and MUMPS 4.9.2 Message-ID: <2045936.HggLLyLscl@mlr114u> Hi, during a timing test I noticed that PETSC3.2/MUMPS4.10.0 takes twice the time to solve a linear system compared to PETSC3.1/MUMPS4.9.2. In both cases I use the same configure options to compile PETSC and the MUMPS was downloaded during the compilation of PETSC. Did someone else get such a difference for the calculation time or can someone tell me what happens? -- Best regards Andreas Hauffe Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" ---------------------------------------------------------------------------------------------------- Technische Universit?t Dresden Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace Engineering Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft Engineering D-01062 Dresden Germany phone : +49 (351) 463 38496 fax : +49 (351) 463 37263 mail : andreas.hauffe at tu-dresden.de Website : http://tu-dresden.de/mw/ilr/lft ---------------------------------------------------------------------------------------------------- From knepley at gmail.com Fri Jun 22 07:26:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Jun 2012 06:26:01 -0600 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: <4FE424BF.8080809@gfz-potsdam.de> References: <4FE1EFAA.3020608@gfz-potsdam.de> <4FE424BF.8080809@gfz-potsdam.de> Message-ID: On Fri, Jun 22, 2012 at 1:54 AM, Alexander Grayver wrote: > Hi Barry, > > In this case the application of the method is quite restricted since all >>> practical least squares problems formulated in form of normal equations are >>> solved with regularization, e.g.: >>> >>> (A'A + \lamba I)x = A'b >>> >> Yes it is restrictive. There is no concept of lambda in CGNE in PETSc >> > > In this case, since there is LSQR in PETSc, there is hardly any reason to > use CGNE. > > Assume I have A computed and use matrix free approach to represent (A'A + >>> \lamba I) without ever forming it, so what should I do then to apply >>> KSPCGNE? >>> >> If you supply a shell matrix that applies (A'A + \lamba I) why not >> just use KSPCG? >> > > That is what I do at the moment. However, as far as I understand, CGLS is > not just about shifting original matrix with some lambda, it has other > advantages over CG for normal equations. > According to the CGLS website, you should use LSQR if lambda > 0. The value of negative shifts is not clear to me. It looks like all these rely on the same trick (bidiagonalization) to avoid squaring the matrix and the condition number. Matt > But if you provide this shell matrix, how do you plan to apply a >> preconditioner? >> > > One can easily compute diag(A'A + \lamba I), thanks to MatGetColumnNorms, > and thus Jacobi is possible. Since the matrix is diagonally dominant, in my > case it is enough to converge. Although convergence for normal equations > does not imply accurate solution, so that one needs CGLS or LSQR. > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 22 07:31:31 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Jun 2012 06:31:31 -0600 Subject: [petsc-users] replacing Schur complement in pcfieldsplit In-Reply-To: References: Message-ID: On Fri, Jun 22, 2012 at 1:46 AM, Klaij, Christiaan wrote: > > > > > > > > > > In order to implement SIMPLE-type preconditioners for the > > > > > incompressible Navier-Stokes equations (Elman e.a. JCP 227, 2008) > > > > > using the PCFieldSplit framework, it looks like I need to replace > > > > > inv(A00) by some cheap approximation > > > > > > > > > > 1) in the Schur complement > > > > > > > > > > > > > When you have a Schur FS, the '0' solver is this approximation. > > > > > > > > > > > > > 2) in the L and/or U of the LDU factorization > > > > > > > > > > > > > You can use whatever PC you want for the solver mentioned above. > > > > > > > > > > > > > 3) while keeping A00 in the D > > > > > > > > > > > > > I think what you want here is -pc_fieldsplit_real_diagonal. > > > > > > Let me get this straight. Looking at the full LDU factorization > > > of the block matrix. Citing from the manual: > > > > > > For the Schur complement preconditioner if > > > J = ( A00 A01 ) > > > ( A10 A11 ) > > > > > > the preconditioner using full factorization is > > > ( I -A10 ksp(A00) ) ( inv(A00) 0 ) ( I 0 ) > > > ( 0 I ) ( 0 ksp(S) ) ( -A10 ksp(A00) I ) > > > > > > > Yes. > > > > > > > Clearly inv(A00) occurs four times, right? In L and in U and > > > twice in D. Now if I somehow overrule the '0' solver with my > > > > > > > Yes > > > > > > > approximation and use -pc_fieldsplit_real_diagonal, the effect > > > would be that inv(A00) is replaced in L, in U and in S but not in > > > the 00-block of D? > > > > > > > No. What this says is that we should use the action of the > > actual matrix rather than the preconditioner matrix in the solver. > > Odd. Don't you *always* need the action of the actual matrix (and > of the preconditioner) in a Krylov subspace method? > No. > > > > I now think I have a better idea what you want, but it would be > > helpful if you wrote it out completely in linear algebra notation, as > > above. Right now, we use the same KSP for all 4 places above. > > Using different KSPs would require a small code change, which I > > can make if you give me a better idea what you want. > > Maybe there is a mistake in the manual, shouldn't it be > > ( I -A01 ksp(A00) ) > ( 0 I ) > Yes, the comments in fieldsplit.c are correct. > in the factorization above instead of -A10 ksp(A00)? SIMPLE-type > preconditioners are usually written as: > > ( I -A01 dA00^(-1) ) ( A00 0 )^(-1) > ( 0 I ) ( A10 S ) > > with S = A11 - A10 dA00^(-1) A01, where dA00^(-1) is the inverse > of the diagonal of A00. Therefore it only requires one solve for > A00 and one solve for S. > Okay, let me think about it. Matt > > > > > > > And what's the function name corresponding to > > > -pc_fieldsplit_real_diagonal? > > > > > > > We have not put one in yet. > > Please let me know when you do. > > > > > Thanks, > > > > Matt > > > dr. ir. Christiaan Klaij > CFD Researcher > Research & Development > E mailto:C.Klaij at marin.nl > T +31 317 49 33 44 > > MARIN > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Jun 22 08:19:01 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 22 Jun 2012 09:19:01 -0400 Subject: [petsc-users] PETSC3.2 and MUMPS 4.10.0 slower than PETSC3.1 and MUMPS 4.9.2 In-Reply-To: <2045936.HggLLyLscl@mlr114u> References: <2045936.HggLLyLscl@mlr114u> Message-ID: My experience is that PETSc 3.3 with MUMPS 4.10.0 is faster than either of these, so I think you should try updating. John On Fri, Jun 22, 2012 at 7:52 AM, Andreas Hauffe wrote: > Hi, > > during a timing test I noticed that PETSC3.2/MUMPS4.10.0 takes twice the time > to solve a linear system compared to PETSC3.1/MUMPS4.9.2. In both cases I use > the same configure options to compile PETSC and the MUMPS was downloaded > during the compilation of PETSC. Did someone else get such a difference for > the calculation time or can someone tell me what happens? > > -- > Best regards > Andreas Hauffe > Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" > > ---------------------------------------------------------------------------------------------------- > Technische Universit?t Dresden > Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace Engineering > Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft Engineering > > D-01062 Dresden > Germany > > phone : +49 (351) 463 38496 > fax : ?+49 (351) 463 37263 > mail : andreas.hauffe at tu-dresden.de > Website : http://tu-dresden.de/mw/ilr/lft > ---------------------------------------------------------------------------------------------------- From gaetano at email.virginia.edu Fri Jun 22 10:03:26 2012 From: gaetano at email.virginia.edu (Gaetano Esposito) Date: Fri, 22 Jun 2012 11:03:26 -0400 Subject: [petsc-users] Compact finite differences Message-ID: Compact finite differences have been already discussed in this post and following responses: http://lists.mcs.anl.gov/pipermail/petsc-users/2011-November/011006.html. If I understand well, the matrix for a 2-D (or 3-D) *implicit* solver that makes use of compact FD for the calculation of derivatives is not "banded" because of the ordering of the x-y-z variables in the solution vector destroys any structure, and the matrix is simply sparse (bandwith=sqrt(n)). In that post is actually suggested that a sparse dense solver could be more attractive because of that. However, if the PDE's are solved explicitly in time, the derivatives in all directions may be calculated independently by using an efficient sequential algorithm (o(n)). I am not familiar with any parallel implementations of the banded diagonal algorithm solvers, but I was wondering whether this could be efficiently implemented in Petsc with the existing MA modules. But at that point, I don't know whether there will be any advantage in possibly using an iterative solver that converges to the exact solution in N iterations for a well behaved matrix as the one deriving from compact FD. --g From knepley at gmail.com Fri Jun 22 11:35:16 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Jun 2012 10:35:16 -0600 Subject: [petsc-users] Compact finite differences In-Reply-To: References: Message-ID: On Fri, Jun 22, 2012 at 9:03 AM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > Compact finite differences have been already discussed in this post > and following responses: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-November/011006.html. > > If I understand well, the matrix for a 2-D (or 3-D) *implicit* solver > that makes use of compact FD for the calculation of derivatives is not > "banded" because of the ordering of the x-y-z variables in the > solution vector destroys any structure, and the matrix is simply > sparse (bandwith=sqrt(n)). In that post is actually suggested that a > sparse dense solver could be more attractive because of that. > The blocks are banded, although in 3D these bands are really wide so I don't think it would make sense anymore to use banded solvers. Matt > However, if the PDE's are solved explicitly in time, the derivatives > in all directions may be calculated independently by using an > efficient sequential algorithm (o(n)). I am not familiar with any > parallel implementations of the banded diagonal algorithm solvers, but > I was wondering whether this could be efficiently implemented in Petsc > with the existing MA modules. But at that point, I don't know whether > there will be any advantage in possibly using an iterative solver that > converges to the exact solution in N iterations for a well behaved > matrix as the one deriving from compact FD. > > --g > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 22 12:19:45 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Jun 2012 12:19:45 -0500 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: <4FE424BF.8080809@gfz-potsdam.de> References: <4FE1EFAA.3020608@gfz-potsdam.de> <4FE424BF.8080809@gfz-potsdam.de> Message-ID: <74D9DC80-6036-4219-A811-C26378339B50@mcs.anl.gov> On Jun 22, 2012, at 2:54 AM, Alexander Grayver wrote: > Hi Barry, > >>> In this case the application of the method is quite restricted since all practical least squares problems formulated in form of normal equations are solved with regularization, e.g.: >>> >>> (A'A + \lamba I)x = A'b >> Yes it is restrictive. There is no concept of lambda in CGNE in PETSc > > In this case, since there is LSQR in PETSc, there is hardly any reason to use CGNE. > >>> Assume I have A computed and use matrix free approach to represent (A'A + \lamba I) without ever forming it, so what should I do then to apply KSPCGNE? >> If you supply a shell matrix that applies (A'A + \lamba I) why not just use KSPCG? > > That is what I do at the moment. However, as far as I understand, CGLS is not just about shifting original matrix with some lambda, it has other advantages over CG for normal equations. Ok, then it is an algorithm that is not currently implemented in PETSc. CGNE is only for people who have A (which is square) and want to solve the normal equations with CG using the preconditioner of A and its transpose for the preconditioner. Basically it allows the user to avoid computing A'A explicitly or making their own shell matrix. It is definitely not a substitute for LSQR. Barry > >> But if you provide this shell matrix, how do you plan to apply a preconditioner? > > One can easily compute diag(A'A + \lamba I), thanks to MatGetColumnNorms, and thus Jacobi is possible. Since the matrix is diagonally dominant, in my case it is enough to converge. Although convergence for normal equations does not imply accurate solution, so that one needs CGLS or LSQR. > > -- > Regards, > Alexander > From popov at uni-mainz.de Fri Jun 22 13:15:45 2012 From: popov at uni-mainz.de (Anton Popov) Date: Fri, 22 Jun 2012 20:15:45 +0200 Subject: [petsc-users] stokes and fieldsplit Message-ID: <4FE4B651.6080300@uni-mainz.de> Hi petsc team, I have a question about using nested matrices and vectors for coupled solution of Stokes system with filedsplit preconditioner. Is there a simple example explaining how I can create nested matrix and vector using "MatCreateNest" and "VecCreateNest" functions, provided that I already have all constituting blocks correctly assembled? I could not find these example online, and if I do it straightforwardly according to documentation, petsc complains that some vector blocks are not setup. Do I need to define velocity and pressure splits using "PCFieldSplitSetIS", because if I don't, petsc says "Unhandled case, must have at least two fields, not 1!"? Do I need to explicitly specify the blocksize, if I want to use the filedsplit preconditioner for the velocity block? A small example would be extremely helpful Thanks a lot, Anton From knepley at gmail.com Fri Jun 22 15:45:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 22 Jun 2012 14:45:55 -0600 Subject: [petsc-users] stokes and fieldsplit In-Reply-To: <4FE4B651.6080300@uni-mainz.de> References: <4FE4B651.6080300@uni-mainz.de> Message-ID: On Fri, Jun 22, 2012 at 12:15 PM, Anton Popov wrote: > Hi petsc team, > I have a question about using nested matrices and vectors for coupled > solution of Stokes system with filedsplit preconditioner. > > Is there a simple example explaining how I can create nested matrix and > vector using > "MatCreateNest" and "VecCreateNest" functions, provided that I already > have all constituting blocks correctly assembled? > I could not find these example online, and if I do it straightforwardly > according to documentation, petsc complains that some vector blocks are not > setup. > You do not need MatNest to use FS. Some of us have a crazy MatNest attachment, so it seems like you do, but it will work with any matrix. > Do I need to define velocity and pressure splits using > "PCFieldSplitSetIS", because if I don't, petsc says "Unhandled case, must > have at least two fields, not 1!"? > If you really have Stokes (a 0 pressure block), use -pc_fieldsplit_detect_saddle_point. > Do I need to explicitly specify the blocksize, if I want to use the > filedsplit preconditioner for the velocity block? > No. > A small example would be extremely helpful > SNES ex62 runs it, but you need my FEM assembly configured to do that. I will post instructions soon. Matt > Thanks a lot, > > Anton > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 22 15:59:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 22 Jun 2012 12:59:58 -0800 Subject: [petsc-users] Compact finite differences In-Reply-To: References: Message-ID: The "mass" matrix for compact FD is usually well-conditioned. I suggest assembling it once so you can experiment with methods. You can try solving it using a Krylov method with only jacobi preconditioning. If that converges fast enough for you, you can reduce memory costs and speed it up somewhat by writing a MatShell that applies the action (instead of using the assembled matrix). On Fri, Jun 22, 2012 at 7:03 AM, Gaetano Esposito < gaetano at email.virginia.edu> wrote: > Compact finite differences have been already discussed in this post > and following responses: > http://lists.mcs.anl.gov/pipermail/petsc-users/2011-November/011006.html. > > If I understand well, the matrix for a 2-D (or 3-D) *implicit* solver > that makes use of compact FD for the calculation of derivatives is not > "banded" because of the ordering of the x-y-z variables in the > solution vector destroys any structure, and the matrix is simply > sparse (bandwith=sqrt(n)). In that post is actually suggested that a > sparse dense solver could be more attractive because of that. > > However, if the PDE's are solved explicitly in time, the derivatives > in all directions may be calculated independently by using an > efficient sequential algorithm (o(n)). I am not familiar with any > parallel implementations of the banded diagonal algorithm solvers, but > I was wondering whether this could be efficiently implemented in Petsc > with the existing MA modules. But at that point, I don't know whether > there will be any advantage in possibly using an iterative solver that > converges to the exact solution in N iterations for a well behaved > matrix as the one deriving from compact FD. > > --g > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 22 16:11:43 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 22 Jun 2012 13:11:43 -0800 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: <74D9DC80-6036-4219-A811-C26378339B50@mcs.anl.gov> References: <4FE1EFAA.3020608@gfz-potsdam.de> <4FE424BF.8080809@gfz-potsdam.de> <74D9DC80-6036-4219-A811-C26378339B50@mcs.anl.gov> Message-ID: On Fri, Jun 22, 2012 at 9:19 AM, Barry Smith wrote: > CGNE is only for people who have A (which is square) and want to solve > the normal equations with CG using the preconditioner of A and its > transpose for the preconditioner. Basically it allows the user to avoid > computing A'A explicitly or making their own shell matrix. It is > definitely not a substitute for LSQR. What? Maybe you are saying the same thing, but CGNE is a general-purpose non-symmetric method. It works well when the singular values are much better behaved than eigenvalues. A unitary matrix is a classic example where CGNE converges in one iteration (unpreconditioned), but GMRES and CGS need N iterations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 22 16:45:42 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 22 Jun 2012 16:45:42 -0500 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: References: <4FE1EFAA.3020608@gfz-potsdam.de> <4FE424BF.8080809@gfz-potsdam.de> <74D9DC80-6036-4219-A811-C26378339B50@mcs.anl.gov> Message-ID: On Jun 22, 2012, at 4:11 PM, Jed Brown wrote: > On Fri, Jun 22, 2012 at 9:19 AM, Barry Smith wrote: > CGNE is only for people who have A (which is square) and want to solve the normal equations with CG using the preconditioner of A and its transpose for the preconditioner. Basically it allows the user to avoid computing A'A explicitly or making their own shell matrix. It is definitely not a substitute for LSQR. > > What? Maybe you are saying the same thing, but CGNE is a general-purpose non-symmetric method. It works well when the singular values are much better behaved than eigenvalues. A unitary matrix is a classic example where CGNE converges in one iteration (unpreconditioned), but GMRES and CGS need N iterations. Maybe you should add this to the KSPCGNE manual page. Without the "What?" :-) Barry From jedbrown at mcs.anl.gov Fri Jun 22 18:23:33 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 22 Jun 2012 15:23:33 -0800 Subject: [petsc-users] Ambiguity of KSPCGNE In-Reply-To: References: <4FE1EFAA.3020608@gfz-potsdam.de> <4FE424BF.8080809@gfz-potsdam.de> <74D9DC80-6036-4219-A811-C26378339B50@mcs.anl.gov> Message-ID: http://petsc.cs.iit.edu/petsc/petsc-dev/rev/90ce2a6ecab0 On Fri, Jun 22, 2012 at 1:45 PM, Barry Smith wrote: > > On Jun 22, 2012, at 4:11 PM, Jed Brown wrote: > > > On Fri, Jun 22, 2012 at 9:19 AM, Barry Smith wrote: > > CGNE is only for people who have A (which is square) and want to solve > the normal equations with CG using the preconditioner of A and its > transpose for the preconditioner. Basically it allows the user to avoid > computing A'A explicitly or making their own shell matrix. It is > definitely not a substitute for LSQR. > > > > What? Maybe you are saying the same thing, but CGNE is a general-purpose > non-symmetric method. It works well when the singular values are much > better behaved than eigenvalues. A unitary matrix is a classic example > where CGNE converges in one iteration (unpreconditioned), but GMRES and CGS > need N iterations. > > Maybe you should add this to the KSPCGNE manual page. Without the > "What?" :-) > > Barry > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Jun 24 13:40:28 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Sun, 24 Jun 2012 23:10:28 +0430 Subject: [petsc-users] Question about Matrix Free Message-ID: Dear developers, I want to define matrix operation used in SNES as Matrix-Free kernel. Here is what I did: MatCreateSNESMF( snes, &JMat ); MatCreateShell( PETSC_COMM_SELF, nt, nt, nt, nt, (void*)&FC, &JMat ); MatShellSetOperation( JMat, MATOP_MULT, (void(*)())&_petsc_matMultFree ); Where FC is my context which saves the flow field data. In the matrix-Free kernel: void solver::_petsc_matMultFree( Mat J, Vec x, Vec y ) { MatShellGetContext( J, &FC ); // perturb the residual ( R -> R(x+eps) ) // do matrix-vector kernel as: y := Mx = Mass/dt + ( R2 - R1 ) / eps } But I received the error at first subspace creation in KSP: [0]PETSC ERROR: MatMult() line 2177 in src/mat/interface/matrix.c [0]PETSC ERROR: PCApplyBAorAB() line 610 in src/ksp/pc/interface/precon.c [0]PETSC ERROR: GMREScycle() line 156 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 3396 in src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 190 in src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c [0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 151 in Newton.cpp Would you please help me find my mistake ? Thanks a lot, BehZad -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jun 24 13:53:20 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Jun 2012 12:53:20 -0600 Subject: [petsc-users] Question about Matrix Free In-Reply-To: References: Message-ID: On Sun, Jun 24, 2012 at 12:40 PM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Dear developers, > > I want to define matrix operation used in SNES as Matrix-Free kernel. Here > is what I did: > > MatCreateSNESMF( snes, &JMat ); > MatCreateShell( PETSC_COMM_SELF, nt, nt, nt, nt, (void*)&FC, &JMat > ); > MatShellSetOperation( JMat, MATOP_MULT, > (void(*)())&_petsc_matMultFree ); > > Where FC is my context which saves the flow field data. In the matrix-Free > kernel: > > void solver::_petsc_matMultFree( Mat J, Vec x, Vec y ) > { > MatShellGetContext( J, &FC ); > // perturb the residual ( R -> R(x+eps) ) > // do matrix-vector kernel as: y := Mx = Mass/dt + ( R2 - R1 ) / eps > } > This does not have the right signature (no return type). Matt > But I received the error at first subspace creation in KSP: > > [0]PETSC ERROR: MatMult() line 2177 in src/mat/interface/matrix.c > [0]PETSC ERROR: PCApplyBAorAB() line 610 in src/ksp/pc/interface/precon.c > [0]PETSC ERROR: GMREScycle() line 156 in src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in > src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: KSPSolve() line 423 in src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 3396 in src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 190 in src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c > [0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 151 in Newton.cpp > > Would you please help me find my mistake ? > > Thanks a lot, > BehZad > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From behzad.baghapour at gmail.com Sun Jun 24 14:34:43 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 25 Jun 2012 00:04:43 +0430 Subject: [petsc-users] Question about Matrix Free In-Reply-To: References: Message-ID: Would you please explain more ? Did you mean the void type of the matrix-vector kernel ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jun 24 18:41:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 24 Jun 2012 17:41:07 -0600 Subject: [petsc-users] Question about Matrix Free In-Reply-To: References: Message-ID: On Sun, Jun 24, 2012 at 1:34 PM, behzad baghapour < behzad.baghapour at gmail.com> wrote: > Would you please explain more ? Did you mean the void type of the > matrix-vector kernel ? > The return type MUST be PetscErrorCode Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Jun 24 19:20:40 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 24 Jun 2012 16:20:40 -0800 Subject: [petsc-users] Question about Matrix Free In-Reply-To: References: Message-ID: On Sun, Jun 24, 2012 at 3:41 PM, Matthew Knepley wrote: > On Sun, Jun 24, 2012 at 1:34 PM, behzad baghapour < > behzad.baghapour at gmail.com> wrote: > >> Would you please explain more ? Did you mean the void type of the >> matrix-vector kernel ? >> > > The return type MUST be PetscErrorCode Also, if you are going to use a C++ class method, you MUST declare it static. Finally, always include the whole error message. The stack trace is a very important part, bet please always paste the whole thing into the email. It is false economy to not include the whole thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.hauffe at tu-dresden.de Mon Jun 25 02:24:42 2012 From: andreas.hauffe at tu-dresden.de (Andreas Hauffe) Date: Mon, 25 Jun 2012 09:24:42 +0200 Subject: [petsc-users] PETSC3.2 and MUMPS 4.10.0 slower than PETSC3.1 and MUMPS 4.9.2 In-Reply-To: References: <2045936.HggLLyLscl@mlr114u> Message-ID: <2921148.0qvLDAF8kz@mlr114u> Wow, you are right. I hadn't done this, because I wanted to wait for slepc 3.3. Andreas Am Freitag, 22. Juni 2012, 09:19:01 schrieb John Fettig: > My experience is that PETSc 3.3 with MUMPS 4.10.0 is faster than > either of these, so I think you should try updating. > > John > > On Fri, Jun 22, 2012 at 7:52 AM, Andreas Hauffe > > wrote: > > Hi, > > > > during a timing test I noticed that PETSC3.2/MUMPS4.10.0 takes twice the > > time to solve a linear system compared to PETSC3.1/MUMPS4.9.2. In both > > cases I use the same configure options to compile PETSC and the MUMPS > > was downloaded during the compilation of PETSC. Did someone else get > > such a difference for the calculation time or can someone tell me what > > happens? > > > > -- > > Best regards > > Andreas Hauffe > > Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" > > > > ------------------------------------------------------------------------ > > ---------------------------- Technische Universit?t Dresden > > Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace > > Engineering Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft > > Engineering > > > > D-01062 Dresden > > Germany > > > > phone : +49 (351) 463 38496 > > fax : +49 (351) 463 37263 > > mail : andreas.hauffe at tu-dresden.de > > Website : http://tu-dresden.de/mw/ilr/lft > > ------------------------------------------------------------------------ > > ---------------------------- -- Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" ---------------------------------------------------------------------------------------------------- Technische Universit?t Dresden Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace Engineering Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft Engineering D-01062 Dresden Germany phone : +49 (351) 463 38496 fax : +49 (351) 463 37263 mail : andreas.hauffe at tu-dresden.de Website : http://tu-dresden.de/mw/ilr/lft ---------------------------------------------------------------------------------------------------- From behzad.baghapour at gmail.com Mon Jun 25 11:03:34 2012 From: behzad.baghapour at gmail.com (behzad baghapour) Date: Mon, 25 Jun 2012 19:33:34 +0330 Subject: [petsc-users] Question about Matrix Free In-Reply-To: References: Message-ID: I see. Thank you. On Mon, Jun 25, 2012 at 3:50 AM, Jed Brown wrote: > On Sun, Jun 24, 2012 at 3:41 PM, Matthew Knepley wrote: > >> On Sun, Jun 24, 2012 at 1:34 PM, behzad baghapour < >> behzad.baghapour at gmail.com> wrote: >> >>> Would you please explain more ? Did you mean the void type of the >>> matrix-vector kernel ? >>> >> >> The return type MUST be PetscErrorCode > > > Also, if you are going to use a C++ class method, you MUST declare it > static. > > > Finally, always include the whole error message. The stack trace is a very > important part, bet please always paste the whole thing into the email. It > is false economy to not include the whole thing. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jun 25 15:25:48 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 25 Jun 2012 15:25:48 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 Message-ID: <4FE8C94C.4040605@gmail.com> Hi, I'm trying to use DMDA to solve my linear equation. It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. It works with 1 cpu. However now, it shows the error below when I use 2 cpu. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. You can override this restriction using -mat_ascii_output_large.! To view the matrix, my command is : call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) call MatView(A_semi,viewer,ierr) call PetscViewerDestroy(viewer,ierr) I tried to use -mat_ascii_output_large but then it just hangs there forever. I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. So how can I view the matrix? Thank you! -- Yours sincerely, TAY wee-beng From jedbrown at mcs.anl.gov Mon Jun 25 15:40:28 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 25 Jun 2012 12:40:28 -0800 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <4FE8C94C.4040605@gmail.com> References: <4FE8C94C.4040605@gmail.com> Message-ID: On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: > Hi, > > I'm trying to use DMDA to solve my linear equation. > > It works fine with 1 cpu. However, problem arise when cpu > 1. I get > segmentation error. > > I tried to use MatView and PetscViewerASCIIOpen to view my matrix to > check if it's correct. > > It works with 1 cpu. However now, it shows the error below when I use 2 > cpu. > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more > than 1024 rows, use binary format instead. > You can override this restriction using -mat_ascii_output_large.! > > To view the matrix, my command is : > > call PetscViewerASCIIOpen(MPI_COMM_**WORLD,"A_semi.txt",viewer,**ierr) > > call MatView(A_semi,viewer,ierr) > > call PetscViewerDestroy(viewer,**ierr) > > I tried to use -mat_ascii_output_large but then it just hangs there > forever. It may just be very slow. Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. > I create the matrix using DMCreateMatrix. I don't have to use > MatCreateAIJ, right? > > Also, I check for the ierr value when I use MatSetValuesStencil to ensure > there's no error. Btw There's no problem with VecView. > > So how can I view the matrix? > > Thank you! > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jun 25 16:18:30 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 25 Jun 2012 16:18:30 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: References: <4FE8C94C.4040605@gmail.com> Message-ID: <4FE8D5A6.6060209@gmail.com> On 25/6/2012 3:40 PM, Jed Brown wrote: > On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng > wrote: > > Hi, > > I'm trying to use DMDA to solve my linear equation. > > It works fine with 1 cpu. However, problem arise when cpu > 1. I > get segmentation error. > > I tried to use MatView and PetscViewerASCIIOpen to view my matrix > to check if it's correct. > > It works with 1 cpu. However now, it shows the error below when I > use 2 cpu. > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: ASCII matrix output not allowed for matrices with > more than 1024 rows, use binary format instead. > You can override this restriction using -mat_ascii_output_large.! > > To view the matrix, my command is : > > call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) > > call MatView(A_semi,viewer,ierr) > > call PetscViewerDestroy(viewer,ierr) > > I tried to use -mat_ascii_output_large but then it just hangs > there forever. > > > It may just be very slow. I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? I also tried to use : call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) but the same error occurs. It just hangs there when 2 cpu are used. > > Why do you need ASCII? Writing anything large in ASCII is hopeless. > Please use binary. > > I create the matrix using DMCreateMatrix. I don't have to use > MatCreateAIJ, right? > > Also, I check for the ierr value when I use MatSetValuesStencil to > ensure there's no error. Btw There's no problem with VecView. > > So how can I view the matrix? > > Thank you! > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Jun 25 16:24:43 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 25 Jun 2012 16:24:43 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <4FE8D5A6.6060209@gmail.com> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> Message-ID: <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: > On 25/6/2012 3:40 PM, Jed Brown wrote: >> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: >> Hi, >> >> I'm trying to use DMDA to solve my linear equation. >> >> It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. >> >> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. It only makes sense to view the matrix in this case for small matrices. You should check the matrix for a very small problem, how are you going to compare the ASCII output to see if it is correct for a huge matrix. Always debug with very small data sets. Barry >> >> It works with 1 cpu. However now, it shows the error below when I use 2 cpu. >> >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Argument out of range! >> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >> You can override this restriction using -mat_ascii_output_large.! >> >> To view the matrix, my command is : >> >> call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) >> >> call MatView(A_semi,viewer,ierr) >> >> call PetscViewerDestroy(viewer,ierr) >> >> I tried to use -mat_ascii_output_large but then it just hangs there forever. >> >> It may just be very slow. > > I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: > > [0]PETSC ERROR: Argument out of range! > [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. > > It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? > > I also tried to use : > > call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) > > but the same error occurs. It just hangs there when 2 cpu are used. > > >> >> Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. >> >> I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? >> >> Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. >> >> So how can I view the matrix? >> >> Thank you! >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> > > From zonexo at gmail.com Mon Jun 25 16:39:55 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 25 Jun 2012 16:39:55 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> Message-ID: <4FE8DAAB.8040105@gmail.com> On 25/6/2012 4:24 PM, Barry Smith wrote: > On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: > >> On 25/6/2012 3:40 PM, Jed Brown wrote: >>> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: >>> Hi, >>> >>> I'm trying to use DMDA to solve my linear equation. >>> >>> It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. >>> >>> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. > It only makes sense to view the matrix in this case for small matrices. You should check the matrix for a very small problem, how are you going to compare the ASCII output to see if it is correct for a huge matrix. Always debug with very small data sets. > > Barry I just want to get a quick view of the matrix. I have the correct matrix generated without using DM. So, I use a file difference comparison software to check the difference, which is rather fast. I just wonder why it takes a long time to view the matrix on screen, or write to file. I guess it should be easy to just view the matrix and spot the error. >>> It works with 1 cpu. However now, it shows the error below when I use 2 cpu. >>> >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Argument out of range! >>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>> You can override this restriction using -mat_ascii_output_large.! >>> >>> To view the matrix, my command is : >>> >>> call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) >>> >>> call MatView(A_semi,viewer,ierr) >>> >>> call PetscViewerDestroy(viewer,ierr) >>> >>> I tried to use -mat_ascii_output_large but then it just hangs there forever. >>> >>> It may just be very slow. >> I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: >> >> [0]PETSC ERROR: Argument out of range! >> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >> >> It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? >> >> I also tried to use : >> >> call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) >> >> but the same error occurs. It just hangs there when 2 cpu are used. >> >> >>> Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. >>> >>> I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? >>> >>> Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. >>> >>> So how can I view the matrix? >>> >>> Thank you! >>> >>> -- >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >> From bsmith at mcs.anl.gov Mon Jun 25 17:57:42 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 25 Jun 2012 17:57:42 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <4FE8DAAB.8040105@gmail.com> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> <4FE8DAAB.8040105@gmail.com> Message-ID: <589BE158-1249-44C6-BB28-CAD48E004C76@mcs.anl.gov> So you run with -mat_ascii_output_large and it hangs? How big is the matrix? If it truly hangs and doesn't just take a long time you can run with -start_in_debugger and then hit control c in the debugger after it has been "hanging" to see where it is. But why not just run with a matrix less than 1024 to check that the matrix is correct. Barry On Jun 25, 2012, at 4:39 PM, TAY wee-beng wrote: > On 25/6/2012 4:24 PM, Barry Smith wrote: >> On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: >> >>> On 25/6/2012 3:40 PM, Jed Brown wrote: >>>> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: >>>> Hi, >>>> >>>> I'm trying to use DMDA to solve my linear equation. >>>> >>>> It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. >>>> >>>> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. >> It only makes sense to view the matrix in this case for small matrices. You should check the matrix for a very small problem, how are you going to compare the ASCII output to see if it is correct for a huge matrix. Always debug with very small data sets. >> >> Barry > I just want to get a quick view of the matrix. I have the correct matrix generated without using DM. So, I use a file difference comparison software to check the difference, which is rather fast. I just wonder why it takes a long time to view the matrix on screen, or write to file. I guess it should be easy to just view the matrix and spot the error. >>>> It works with 1 cpu. However now, it shows the error below when I use 2 cpu. >>>> >>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [0]PETSC ERROR: Argument out of range! >>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>>> You can override this restriction using -mat_ascii_output_large.! >>>> >>>> To view the matrix, my command is : >>>> >>>> call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) >>>> >>>> call MatView(A_semi,viewer,ierr) >>>> >>>> call PetscViewerDestroy(viewer,ierr) >>>> >>>> I tried to use -mat_ascii_output_large but then it just hangs there forever. >>>> >>>> It may just be very slow. >>> I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: >>> >>> [0]PETSC ERROR: Argument out of range! >>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>> >>> It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? >>> >>> I also tried to use : >>> >>> call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) >>> >>> but the same error occurs. It just hangs there when 2 cpu are used. >>> >>> >>>> Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. >>>> I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? >>>> >>>> Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. >>>> >>>> So how can I view the matrix? >>>> >>>> Thank you! >>>> >>>> -- >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>> > > From hzhang at mcs.anl.gov Mon Jun 25 20:13:30 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 25 Jun 2012 20:13:30 -0500 Subject: [petsc-users] PETSC3.2 and MUMPS 4.10.0 slower than PETSC3.1 and MUMPS 4.9.2 In-Reply-To: <2921148.0qvLDAF8kz@mlr114u> References: <2045936.HggLLyLscl@mlr114u> <2921148.0qvLDAF8kz@mlr114u> Message-ID: You may try slepc-dev http://www.grycap.upv.es/slepc/download/download.htm Hong On Mon, Jun 25, 2012 at 2:24 AM, Andreas Hauffe < andreas.hauffe at tu-dresden.de> wrote: > Wow, you are right. I hadn't done this, because I wanted to wait for slepc > 3.3. > > Andreas > > Am Freitag, 22. Juni 2012, 09:19:01 schrieb John Fettig: > > My experience is that PETSc 3.3 with MUMPS 4.10.0 is faster than > > either of these, so I think you should try updating. > > > > John > > > > On Fri, Jun 22, 2012 at 7:52 AM, Andreas Hauffe > > > > wrote: > > > Hi, > > > > > > during a timing test I noticed that PETSC3.2/MUMPS4.10.0 takes twice > the > > > time to solve a linear system compared to PETSC3.1/MUMPS4.9.2. In both > > > cases I use the same configure options to compile PETSC and the MUMPS > > > was downloaded during the compilation of PETSC. Did someone else get > > > such a difference for the calculation time or can someone tell me what > > > happens? > > > > > > -- > > > Best regards > > > Andreas Hauffe > > > Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" > > > > > > > ------------------------------------------------------------------------ > > > ---------------------------- Technische Universit?t Dresden > > > Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace > > > Engineering Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft > > > Engineering > > > > > > D-01062 Dresden > > > Germany > > > > > > phone : +49 (351) 463 38496 > > > fax : +49 (351) 463 37263 > > > mail : andreas.hauffe at tu-dresden.de > > > Website : http://tu-dresden.de/mw/ilr/lft > > > > ------------------------------------------------------------------------ > > > ---------------------------- > -- > Leiter der Arbeitsgruppe "Auslegungsmethoden f?r Luftfahrzeuge" > > > ---------------------------------------------------------------------------------------------------- > Technische Universit?t Dresden > Institut f?r Luft- und Raumfahrttechnik / Institute of Aerospace > Engineering > Lehrstuhl f?r Luftfahrzeugtechnik / Chair of Aircraft Engineering > > D-01062 Dresden > Germany > > phone : +49 (351) 463 38496 > fax : +49 (351) 463 37263 > mail : andreas.hauffe at tu-dresden.de > Website : http://tu-dresden.de/mw/ilr/lft > > ---------------------------------------------------------------------------------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Jun 25 21:44:21 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 25 Jun 2012 21:44:21 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <589BE158-1249-44C6-BB28-CAD48E004C76@mcs.anl.gov> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> <4FE8DAAB.8040105@gmail.com> <589BE158-1249-44C6-BB28-CAD48E004C76@mcs.anl.gov> Message-ID: <4FE92205.2060208@gmail.com> On 25/6/2012 5:57 PM, Barry Smith wrote: > So you run with -mat_ascii_output_large and it hangs? How big is the matrix? > > If it truly hangs and doesn't just take a long time you can run with -start_in_debugger and then hit control c in the debugger after it has been "hanging" to see where it is. > > But why not just run with a matrix less than 1024 to check that the matrix is correct. > > Barry I have switched to a 2d case. The matrix size is much smaller (still more than 1024). The matrix is now saved to a txt file. However, when running 2 cpu, I got: Matrix Object: 1 MPI processes type: mpiaij row 0: (0, -2) (132, 3) (264, -1) row 1: (1, -2) (133, 3) (265, -1) row 2: (2, -2) (134, 3) (266, -1) ... Shouldn't it be 2 MPI processes? Like in a vector: Vector Object:Vec_84000004_0 2 MPI processes type: mpi Process [0] 0 0 ... I used : call PetscViewerASCIIOpen(MPI_COMM_WORLD,'A_semi.txt',viewer,ierr) call MatView(A_semi,viewer,ierr) call PetscViewerDestroy(viewer,ierr) Is there some error in my matrix? I also noticed that the matrix creating w/o using DM gives the same "1 MPI processes" output. I create it using : call DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& PETSC_DECIDE,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) call DMCreateMatrix(da,MATAIJ,A_semi,ierr) Thanks! > > On Jun 25, 2012, at 4:39 PM, TAY wee-beng wrote: > >> On 25/6/2012 4:24 PM, Barry Smith wrote: >>> On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: >>> >>>> On 25/6/2012 3:40 PM, Jed Brown wrote: >>>>> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: >>>>> Hi, >>>>> >>>>> I'm trying to use DMDA to solve my linear equation. >>>>> >>>>> It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. >>>>> >>>>> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. >>> It only makes sense to view the matrix in this case for small matrices. You should check the matrix for a very small problem, how are you going to compare the ASCII output to see if it is correct for a huge matrix. Always debug with very small data sets. >>> >>> Barry >> I just want to get a quick view of the matrix. I have the correct matrix generated without using DM. So, I use a file difference comparison software to check the difference, which is rather fast. I just wonder why it takes a long time to view the matrix on screen, or write to file. I guess it should be easy to just view the matrix and spot the error. >>>>> It works with 1 cpu. However now, it shows the error below when I use 2 cpu. >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Argument out of range! >>>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>>>> You can override this restriction using -mat_ascii_output_large.! >>>>> >>>>> To view the matrix, my command is : >>>>> >>>>> call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) >>>>> >>>>> call MatView(A_semi,viewer,ierr) >>>>> >>>>> call PetscViewerDestroy(viewer,ierr) >>>>> >>>>> I tried to use -mat_ascii_output_large but then it just hangs there forever. >>>>> >>>>> It may just be very slow. >>>> I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: >>>> >>>> [0]PETSC ERROR: Argument out of range! >>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>>> >>>> It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? >>>> >>>> I also tried to use : >>>> >>>> call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) >>>> >>>> but the same error occurs. It just hangs there when 2 cpu are used. >>>> >>>> >>>>> Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. >>>>> I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? >>>>> >>>>> Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. >>>>> >>>>> So how can I view the matrix? >>>>> >>>>> Thank you! >>>>> >>>>> -- >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >> From bsmith at mcs.anl.gov Mon Jun 25 21:46:50 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 25 Jun 2012 21:46:50 -0500 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <4FE92205.2060208@gmail.com> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> <4FE8DAAB.8040105@gmail.com> <589BE158-1249-44C6-BB28-CAD48E004C76@mcs.anl.gov> <4FE92205.2060208@gmail.com> Message-ID: <0635EA3E-1860-4973-9313-EFECA7BCFA18@mcs.anl.gov> It prints it in the natural ordering and does not print the division of the matrix into its two parts for each process. Barry On Jun 25, 2012, at 9:44 PM, TAY wee-beng wrote: > On 25/6/2012 5:57 PM, Barry Smith wrote: >> So you run with -mat_ascii_output_large and it hangs? How big is the matrix? >> >> If it truly hangs and doesn't just take a long time you can run with -start_in_debugger and then hit control c in the debugger after it has been "hanging" to see where it is. >> >> But why not just run with a matrix less than 1024 to check that the matrix is correct. >> >> Barry > > I have switched to a 2d case. The matrix size is much smaller (still more than 1024). The matrix is now saved to a txt file. However, when running 2 cpu, I got: > > Matrix Object: 1 MPI processes > type: mpiaij > row 0: (0, -2) (132, 3) (264, -1) > row 1: (1, -2) (133, 3) (265, -1) > row 2: (2, -2) (134, 3) (266, -1) > ... > > Shouldn't it be 2 MPI processes? Like in a vector: > > Vector Object:Vec_84000004_0 2 MPI processes > type: mpi > Process [0] > 0 > 0 > ... > > I used : > > call PetscViewerASCIIOpen(MPI_COMM_WORLD,'A_semi.txt',viewer,ierr) > > call MatView(A_semi,viewer,ierr) > > call PetscViewerDestroy(viewer,ierr) > > Is there some error in my matrix? I also noticed that the matrix creating w/o using DM gives the same "1 MPI processes" output. > > I create it using : > > call DMDACreate2d(MPI_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,size_x,size_y,& > PETSC_DECIDE,num_procs,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr) > > call DMCreateMatrix(da,MATAIJ,A_semi,ierr) > > Thanks! > > >> >> On Jun 25, 2012, at 4:39 PM, TAY wee-beng wrote: >> >>> On 25/6/2012 4:24 PM, Barry Smith wrote: >>>> On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: >>>> >>>>> On 25/6/2012 3:40 PM, Jed Brown wrote: >>>>>> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng wrote: >>>>>> Hi, >>>>>> >>>>>> I'm trying to use DMDA to solve my linear equation. >>>>>> >>>>>> It works fine with 1 cpu. However, problem arise when cpu > 1. I get segmentation error. >>>>>> >>>>>> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to check if it's correct. >>>> It only makes sense to view the matrix in this case for small matrices. You should check the matrix for a very small problem, how are you going to compare the ASCII output to see if it is correct for a huge matrix. Always debug with very small data sets. >>>> >>>> Barry >>> I just want to get a quick view of the matrix. I have the correct matrix generated without using DM. So, I use a file difference comparison software to check the difference, which is rather fast. I just wonder why it takes a long time to view the matrix on screen, or write to file. I guess it should be easy to just view the matrix and spot the error. >>>>>> It works with 1 cpu. However now, it shows the error below when I use 2 cpu. >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>>> [0]PETSC ERROR: Argument out of range! >>>>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>>>>> You can override this restriction using -mat_ascii_output_large.! >>>>>> >>>>>> To view the matrix, my command is : >>>>>> >>>>>> call PetscViewerASCIIOpen(MPI_COMM_WORLD,"A_semi.txt",viewer,ierr) >>>>>> >>>>>> call MatView(A_semi,viewer,ierr) >>>>>> >>>>>> call PetscViewerDestroy(viewer,ierr) >>>>>> >>>>>> I tried to use -mat_ascii_output_large but then it just hangs there forever. >>>>>> >>>>>> It may just be very slow. >>>>> I tried using 1 cpu and I got it in a few minutes. Should i take a long time if I use 2 cpu? Also, I got the error: >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more than 1024 rows, use binary format instead. >>>>> >>>>> It doesn't happen when I use 1 cpu. Is it normal to get it when I use 2 cpu? >>>>> >>>>> I also tried to use : >>>>> >>>>> call MatView(A_semi,PETSC_VIEWER_STDOUT_WORLD ,ierr) >>>>> >>>>> but the same error occurs. It just hangs there when 2 cpu are used. >>>>> >>>>> >>>>>> Why do you need ASCII? Writing anything large in ASCII is hopeless. Please use binary. >>>>>> I create the matrix using DMCreateMatrix. I don't have to use MatCreateAIJ, right? >>>>>> >>>>>> Also, I check for the ierr value when I use MatSetValuesStencil to ensure there's no error. Btw There's no problem with VecView. >>>>>> >>>>>> So how can I view the matrix? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> -- >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> >>> > > From jedbrown at mcs.anl.gov Mon Jun 25 21:49:34 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 25 Jun 2012 18:49:34 -0800 Subject: [petsc-users] MatView and PetscViewerASCIIOpen problem when cpu > 1 In-Reply-To: <4FE92205.2060208@gmail.com> References: <4FE8C94C.4040605@gmail.com> <4FE8D5A6.6060209@gmail.com> <6D738F52-2A57-4765-ABB2-5D77F0DA1676@mcs.anl.gov> <4FE8DAAB.8040105@gmail.com> <589BE158-1249-44C6-BB28-CAD48E004C76@mcs.anl.gov> <4FE92205.2060208@gmail.com> Message-ID: That particular issue is an output diagnostic quirk that will be fixed eventually. Don't worry about it now. On Jun 25, 2012 6:44 PM, "TAY wee-beng" wrote: > On 25/6/2012 5:57 PM, Barry Smith wrote: > >> So you run with -mat_ascii_output_large and it hangs? How big is the >> matrix? >> >> If it truly hangs and doesn't just take a long time you can run with >> -start_in_debugger and then hit control c in the debugger after it has been >> "hanging" to see where it is. >> >> But why not just run with a matrix less than 1024 to check that the >> matrix is correct. >> >> Barry >> > > I have switched to a 2d case. The matrix size is much smaller (still more > than 1024). The matrix is now saved to a txt file. However, when running 2 > cpu, I got: > > Matrix Object: 1 MPI processes > type: mpiaij > row 0: (0, -2) (132, 3) (264, -1) > row 1: (1, -2) (133, 3) (265, -1) > row 2: (2, -2) (134, 3) (266, -1) > ... > > Shouldn't it be 2 MPI processes? Like in a vector: > > Vector Object:Vec_84000004_0 2 MPI processes > type: mpi > Process [0] > 0 > 0 > ... > > I used : > > call PetscViewerASCIIOpen(MPI_COMM_**WORLD,'A_semi.txt',viewer,**ierr) > > call MatView(A_semi,viewer,ierr) > > call PetscViewerDestroy(viewer,**ierr) > > Is there some error in my matrix? I also noticed that the matrix creating > w/o using DM gives the same "1 MPI processes" output. > > I create it using : > > call DMDACreate2d(MPI_COMM_WORLD,**DMDA_BOUNDARY_NONE,DMDA_** > BOUNDARY_NONE,DMDA_STENCIL_**STAR,size_x,size_y,& > PETSC_DECIDE,num_procs,i1,i1,**PETSC_NULL_INTEGER,PETSC_NULL_** > INTEGER,da,ierr) > > call DMCreateMatrix(da,MATAIJ,A_**semi,ierr) > > Thanks! > > > >> On Jun 25, 2012, at 4:39 PM, TAY wee-beng wrote: >> >> On 25/6/2012 4:24 PM, Barry Smith wrote: >>> >>>> On Jun 25, 2012, at 4:18 PM, TAY wee-beng wrote: >>>> >>>> On 25/6/2012 3:40 PM, Jed Brown wrote: >>>>> >>>>>> On Mon, Jun 25, 2012 at 12:25 PM, TAY wee-beng >>>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> I'm trying to use DMDA to solve my linear equation. >>>>>> >>>>>> It works fine with 1 cpu. However, problem arise when cpu > 1. I get >>>>>> segmentation error. >>>>>> >>>>>> I tried to use MatView and PetscViewerASCIIOpen to view my matrix to >>>>>> check if it's correct. >>>>>> >>>>> It only makes sense to view the matrix in this case for small >>>> matrices. You should check the matrix for a very small problem, how are you >>>> going to compare the ASCII output to see if it is correct for a huge >>>> matrix. Always debug with very small data sets. >>>> >>>> Barry >>>> >>> I just want to get a quick view of the matrix. I have the correct matrix >>> generated without using DM. So, I use a file difference comparison >>> software to check the difference, which is rather fast. I just wonder why >>> it takes a long time to view the matrix on screen, or write to file. I >>> guess it should be easy to just view the matrix and spot the error. >>> >>>> It works with 1 cpu. However now, it shows the error below when I use 2 >>>>>> cpu. >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------**------ >>>>>> [0]PETSC ERROR: Argument out of range! >>>>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with >>>>>> more than 1024 rows, use binary format instead. >>>>>> You can override this restriction using -mat_ascii_output_large.! >>>>>> >>>>>> To view the matrix, my command is : >>>>>> >>>>>> call PetscViewerASCIIOpen(MPI_COMM_**WORLD,"A_semi.txt",viewer,** >>>>>> ierr) >>>>>> >>>>>> call MatView(A_semi,viewer,ierr) >>>>>> >>>>>> call PetscViewerDestroy(viewer,**ierr) >>>>>> >>>>>> I tried to use -mat_ascii_output_large but then it just hangs there >>>>>> forever. >>>>>> >>>>>> It may just be very slow. >>>>>> >>>>> I tried using 1 cpu and I got it in a few minutes. Should i take a >>>>> long time if I use 2 cpu? Also, I got the error: >>>>> >>>>> [0]PETSC ERROR: Argument out of range! >>>>> [0]PETSC ERROR: ASCII matrix output not allowed for matrices with more >>>>> than 1024 rows, use binary format instead. >>>>> >>>>> It doesn't happen when I use 1 cpu. Is it normal to get it when I use >>>>> 2 cpu? >>>>> >>>>> I also tried to use : >>>>> >>>>> call MatView(A_semi,PETSC_VIEWER_**STDOUT_WORLD ,ierr) >>>>> >>>>> but the same error occurs. It just hangs there when 2 cpu are used. >>>>> >>>>> >>>>> Why do you need ASCII? Writing anything large in ASCII is hopeless. >>>>>> Please use binary. >>>>>> I create the matrix using DMCreateMatrix. I don't have to use >>>>>> MatCreateAIJ, right? >>>>>> >>>>>> Also, I check for the ierr value when I use MatSetValuesStencil to >>>>>> ensure there's no error. Btw There's no problem with VecView. >>>>>> >>>>>> So how can I view the matrix? >>>>>> >>>>>> Thank you! >>>>>> >>>>>> -- >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> >>>>>> >>> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Tue Jun 26 12:37:39 2012 From: mrosso at uci.edu (Michele Rosso) Date: Tue, 26 Jun 2012 10:37:39 -0700 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver Message-ID: <4FE9F363.8040506@uci.edu> Hi, I need some help to use the PETSc library inside my Fortran 95 code. My goal is to solve a symmetric linear system that derives from the finite difference discretization of the Poisson equation. I will use the preconditioned conjugate method. I am mostly interested in how to decompose my data among the different processes. In particular: 1) Since my code already implements the 2D-decomposition, would it be best to build the matrix with the DMDA object type, DA object type or the regular Mat type? 2) After inserting values into a vector/matrix, PETSc performs any needed message passing of nonlocal components, thus the values locally contained on a process may be communicated to another process. How can I revert this at the end of the computation, that is, how can I be sure that the local solution vector contains the values associated to the grid nodes contained into the hosting process? Thank you, Michele -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 26 12:42:24 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 09:42:24 -0800 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver In-Reply-To: <4FE9F363.8040506@uci.edu> References: <4FE9F363.8040506@uci.edu> Message-ID: On Tue, Jun 26, 2012 at 9:37 AM, Michele Rosso wrote: > Hi, > > I need some help to use the PETSc library inside my Fortran 95 code. > My goal is to solve a symmetric linear system that derives from the finite > difference discretization > of the Poisson equation. I will use the preconditioned conjugate method. > > I am mostly interested in how to decompose my data among the different > processes. > In particular: > > 1) Since my code already implements the 2D-decomposition, would it be best > to build the matrix with the DMDA object type, DA object type > or the regular Mat type? > It is certainly easiest to just use DMDA (and you will get geometric multigrid for free, which is unbeatable for this problem), but you can work directly with Mat if you prefer. See src/ksp/ksp/examples/tutorials/ex45f.F for a Fortran example. > > 2) After inserting values into a vector/matrix, PETSc performs any needed > message passing of nonlocal components, thus the values locally contained > on a process may be communicated to another process. How can I revert > this at the end of the computation, that is, how can I be sure that the > local solution vector contains the values associated to the grid nodes > contained into the hosting process? > Please read the section of the user's manual on local versus global spaces and the structured grid decompositions. If you use DM, there is DMGlobalToLocalBegin/End that update the ghost points in the local vectors. > > > Thank you, > > Michele > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Tue Jun 26 13:24:45 2012 From: mrosso at uci.edu (Michele Rosso) Date: Tue, 26 Jun 2012 11:24:45 -0700 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver In-Reply-To: References: <4FE9F363.8040506@uci.edu> Message-ID: <4FE9FE6D.5090905@uci.edu> Thanks Jed for your reply. 1) I will use the DMDA object type then. I am still not very clear about the difference between DA and DMDA though. 2) I am not interested in having ghost nodes updated. I want the proper values of the solution on the proper processor. I have to fill the known-terms-vector with nodes-dependent values ( in contrast with ex45f.F, where vector b is filled with 1s, thus there is no dependence on the grid location). Since every processor defines "non-local" (according to the PETSc internal ordering) components, the vector is re-arranged and so is the solution vector. So I will have on every process a solution which partially should be on a difference process. And this is not about ghost cell. Sorry for this long explanation but I am trying to be as clear as I can. Thank you fro your help and patience, Michele On 06/26/2012 10:42 AM, Jed Brown wrote: > On Tue, Jun 26, 2012 at 9:37 AM, Michele Rosso > wrote: > > Hi, > > I need some help to use the PETSc library inside my Fortran 95 code. > My goal is to solve a symmetric linear system that derives from > the finite difference discretization > of the Poisson equation. I will use the preconditioned conjugate > method. > > I am mostly interested in how to decompose my data among the > different processes. > In particular: > > 1) Since my code already implements the 2D-decomposition, would it > be best to build the matrix with the DMDA object type, DA object type > or the regular Mat type? > > > It is certainly easiest to just use DMDA (and you will get geometric > multigrid for free, which is unbeatable for this problem), but you can > work directly with Mat if you prefer. See > src/ksp/ksp/examples/tutorials/ex45f.F for a Fortran example. > > > 2) After inserting values into a vector/matrix, PETSc performs any > needed message passing of nonlocal components, thus the values > locally contained on a process may be communicated to another > process. How can I revert this at the end of the computation, > that is, how can I be sure that the local solution vector contains > the values associated to the grid nodes contained into the hosting > process? > > > Please read the section of the user's manual on local versus global > spaces and the structured grid decompositions. If you use DM, there is > DMGlobalToLocalBegin/End that update the ghost points in the local > vectors. > > > > Thank you, > > Michele > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 26 13:36:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 10:36:13 -0800 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver In-Reply-To: <4FE9FE6D.5090905@uci.edu> References: <4FE9F363.8040506@uci.edu> <4FE9FE6D.5090905@uci.edu> Message-ID: On Tue, Jun 26, 2012 at 10:24 AM, Michele Rosso wrote: > Thanks Jed for your reply. > > 1) I will use the DMDA object type then. I am still not very clear about > the difference between DA and DMDA though. > DM is the generic interface, DMDA is the structured grid implementation. It used to be called just DA and managed a bit different. Maybe you were looking at old docs? > > 2) I am not interested in having ghost nodes updated. I want the proper > values of the solution on the proper processor. > I have to fill the known-terms-vector with nodes-dependent values ( > in contrast with ex45f.F, where vector b is filled with 1s, thus there is > no dependence on the grid location). Since every processor defines > "non-local" (according to the PETSc internal ordering) components, the > vector is re-arranged > and so is the solution vector. So I will have on every process a > solution which partially should be on a difference process. And this is > not about ghost cell. > Sorry for this long explanation but I am trying to be as clear as I > can. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGlobalToNaturalBegin.html > > Thank you fro your help and patience, > > Michele > > > > > > > > > > On 06/26/2012 10:42 AM, Jed Brown wrote: > > On Tue, Jun 26, 2012 at 9:37 AM, Michele Rosso wrote: > >> Hi, >> >> I need some help to use the PETSc library inside my Fortran 95 code. >> My goal is to solve a symmetric linear system that derives from the >> finite difference discretization >> of the Poisson equation. I will use the preconditioned conjugate method. >> >> I am mostly interested in how to decompose my data among the different >> processes. >> In particular: >> >> 1) Since my code already implements the 2D-decomposition, would it be >> best to build the matrix with the DMDA object type, DA object type >> or the regular Mat type? >> > > It is certainly easiest to just use DMDA (and you will get geometric > multigrid for free, which is unbeatable for this problem), but you can work > directly with Mat if you prefer. See src/ksp/ksp/examples/tutorials/ex45f.F > for a Fortran example. > > >> >> 2) After inserting values into a vector/matrix, PETSc performs any needed >> message passing of nonlocal components, thus the values locally contained >> on a process may be communicated to another process. How can I revert >> this at the end of the computation, that is, how can I be sure that the >> local solution vector contains the values associated to the grid nodes >> contained into the hosting process? >> > > Please read the section of the user's manual on local versus global > spaces and the structured grid decompositions. If you use DM, there is > DMGlobalToLocalBegin/End that update the ghost points in the local vectors. > > >> >> >> Thank you, >> >> Michele >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paeanball at gmail.com Tue Jun 26 14:22:40 2012 From: paeanball at gmail.com (Bao Kai) Date: Tue, 26 Jun 2012 22:22:40 +0300 Subject: [petsc-users] MatFDColorCreate takes really big portion of the total time Message-ID: Hi, all, I use the SNES in petsc-3.2 to solve my problem. The problem is a 3-D finite difference problem with structured grid. I use MatFDColorCreate to generate the Jacobian matrix. I just found that when the size of problem is big, MatFDColorCreate takes really long time. The following results is the summary with size of the mesh to be 1000^3. 90% of the time is costed in MatFDColorCreate. 237 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 238 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 239 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 240 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 241 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 And the following the code I use. 262 call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr) 263 call DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr) 264 call MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr) 265 call MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr) 266 call MatFDColoringSetFromOptions(matfdcoloring,ierr) 267 call SNESSetJacobian(solv%snes, solv%jac, solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr) 268 call ISColoringDestroy(iscoloring,ierr) I am wondering if there is anything I can do to improve this problem. Thank you very much. Best Regards, Kai From bsmith at mcs.anl.gov Tue Jun 26 14:27:28 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 26 Jun 2012 14:27:28 -0500 Subject: [petsc-users] MatFDColorCreate takes really big portion of the total time In-Reply-To: References: Message-ID: Parallel or sequential? Barry Please send entire -log_summary output On Jun 26, 2012, at 2:22 PM, Bao Kai wrote: > Hi, all, > > I use the SNES in petsc-3.2 to solve my problem. The problem is a 3-D > finite difference problem with structured grid. > > I use MatFDColorCreate to generate the Jacobian matrix. I just found > that when the size of problem is big, MatFDColorCreate takes really > long time. The following results is the summary with size of the mesh > to be 1000^3. 90% of the time is costed in MatFDColorCreate. > > 237 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 > 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 238 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 > 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 239 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 > 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 > 240 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 > 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 > 241 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 > 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 > > And the following the code I use. > > 262 call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr) > 263 call > DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr) > 264 call MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr) > 265 call > MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr) > 266 call MatFDColoringSetFromOptions(matfdcoloring,ierr) > 267 call SNESSetJacobian(solv%snes, solv%jac, > solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr) > 268 call ISColoringDestroy(iscoloring,ierr) > > I am wondering if there is anything I can do to improve this problem. > > Thank you very much. > > Best Regards, > Kai From paeanball at gmail.com Tue Jun 26 14:37:47 2012 From: paeanball at gmail.com (Bao Kai) Date: Tue, 26 Jun 2012 22:37:47 +0300 Subject: [petsc-users] MatFDColorCreate takes really big portion of the total time Message-ID: Hi, Barry, Parallel on a bluegene/P machine. Attached please find the result information. The log_summary output is at the bottom of the file. BTW: how to respond in the mailing list directly? I always have to copy the message in my mailbox to reply. Best Regards, Kai Parallel or sequential? Barry Please send entire -log_summary output On Jun 26, 2012, at 2:22 PM, Bao Kai wrote: > Hi, all, > > I use the SNES in petsc-3.2 to solve my problem. The problem is a 3-D > finite difference problem with structured grid. > > I use MatFDColorCreate to generate the Jacobian matrix. I just found > that when the size of problem is big, MatFDColorCreate takes really > long time. The following results is the summary with size of the mesh > to be 1000^3. 90% of the time is costed in MatFDColorCreate. > > 237 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 > 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 238 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 > 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 239 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 > 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 > 240 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 > 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 > 241 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 > 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 > > And the following the code I use. > > 262 call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr) > 263 call > DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr) > 264 call MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr) > 265 call > MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr) > 266 call MatFDColoringSetFromOptions(matfdcoloring,ierr) > 267 call SNESSetJacobian(solv%snes, solv%jac, > solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr) > 268 call ISColoringDestroy(iscoloring,ierr) > > I am wondering if there is anything I can do to improve this problem. > > Thank you very much. > > Best Regards, > Kai -------------- next part -------------- A non-text attachment was scrubbed... Name: results_32768.out Type: application/octet-stream Size: 22976 bytes Desc: not available URL: From paeanball at gmail.com Tue Jun 26 14:38:53 2012 From: paeanball at gmail.com (Bao Kai) Date: Tue, 26 Jun 2012 22:38:53 +0300 Subject: [petsc-users] MatFDColorCreate takes really big portion of the total time In-Reply-To: References: Message-ID: Sorry, I failed in attaching file. Kai On 6/26/12, Bao Kai wrote: > Hi, Barry, > > Parallel on a bluegene/P machine. > > Attached please find the result information. The log_summary output is > at the bottom of the file. > > > BTW: how to respond in the mailing list directly? I always have to > copy the message in my mailbox to reply. > > Best Regards, > Kai > > > > Parallel or sequential? > > Barry > > Please send entire -log_summary output > > On Jun 26, 2012, at 2:22 PM, Bao Kai wrote: > >> Hi, all, >> >> I use the SNES in petsc-3.2 to solve my problem. The problem is a 3-D >> finite difference problem with structured grid. >> >> I use MatFDColorCreate to generate the Jacobian matrix. I just found >> that when the size of problem is big, MatFDColorCreate takes really >> long time. The following results is the summary with size of the mesh >> to be 1000^3. 90% of the time is costed in MatFDColorCreate. >> >> 237 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 >> 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> 238 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 >> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> 239 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 >> 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 >> 240 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 >> 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 >> 241 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 >> 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 >> >> And the following the code I use. >> >> 262 call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr) >> 263 call >> DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr) >> 264 call >> MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr) >> 265 call >> MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr) >> 266 call MatFDColoringSetFromOptions(matfdcoloring,ierr) >> 267 call SNESSetJacobian(solv%snes, solv%jac, >> solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr) >> 268 call ISColoringDestroy(iscoloring,ierr) >> >> I am wondering if there is anything I can do to improve this problem. >> >> Thank you very much. >> >> Best Regards, >> Kai > -------------- next part -------------- A non-text attachment was scrubbed... Name: results_32768.out Type: application/octet-stream Size: 22976 bytes Desc: not available URL: From paulm at txcorp.com Tue Jun 26 15:29:00 2012 From: paulm at txcorp.com (Paul Mullowney) Date: Tue, 26 Jun 2012 14:29:00 -0600 Subject: [petsc-users] potential problem in dageometry Message-ID: <4FEA1B8C.2040105@txcorp.com> Hi All, I'm trying to build PETSc with the configuration: ./configure -with-debugging=no --with-mpi=1 --with-mpi-dir=/usr/mpi/gcc/openmpi-1.4.3 --CFLAGS="-g -O3" --with-precision=double --with-clanguage=C++ --with-scalar-type=complex --with-thrust=0 --with-cusp=0 --with-cuda=0 I see the errors listed below. I'm wondering if the code in DMDAComputeCellGeometry_2D function should be using PetscReal instead of PetscScalar? Thanks, -Paul dageometry.c: In function ?PetscErrorCode DMDAComputeCellGeometry_2D(_p_DM*, const PetscScalar*, const PetscReal*, PetscReal*, PetscReal*, PetscScalar*)?: dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:323: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:324: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:324: warning: cannot pass objects of non-POD type ?const struct PetscScalar? through ?...?; call will abort at runtime dageometry.c:325: error: cannot convert ?std::complex? to ?double? in assignment dageometry.c:325: error: cannot convert ?std::complex? to ?double? in assignment dageometry.c:326: error: cannot convert ?std::complex? to ?double? in assignment dageometry.c:326: error: cannot convert ?std::complex? to ?double? in assignment dageometry.c:328: error: cannot convert ?std::complex? to ?PetscReal? in assignment dageometry.c: In function ?PetscErrorCode DMDAComputeCellGeometry(_p_DM*, PetscInt, PetscQuadrature*, PetscReal*, PetscReal*, PetscReal*, PetscReal*)?: dageometry.c:352: error: cannot convert ?const std::complex? to ?double? in assignment dageometry.c:357: error: cannot convert ?PetscReal*? to ?PetscScalar*? for argument ?6? to ?PetscErrorCode DMDAComputeCellGeometry_2D(_p_DM*, const PetscScalar*, const PetscReal*, PetscReal*, PetscReal*, PetscScalar*)? /usr/bin/ar: dageometry.o: No such file or directory From Shuangshuang.Jin at pnnl.gov Tue Jun 26 16:09:01 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 26 Jun 2012 14:09:01 -0700 Subject: [petsc-users] Loading Binary file problem Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018F1929B8E0@EMAIL04.pnl.gov> Hello everyone, I created a 4*3 matrix A in matlab, saved it to binary file "myA.dat" using PetscBinaryWrite('myA.dat',sparse(A)). Everything seems fine to me because I can read the binary file 'myA.dat' by PetscBinaryRead('myA.dat')correctly. A = [1 2 0; 2 0 5; 0 3 0; 1 4 1] PetscBinaryWrite('myA.dat',sparse(A)) PetscBinaryRead('myA.dat') However, when I tried to load the binary file "myA.dat" in PETSc using the example code "ex12.c" under "petsc-3.3-p0/src/mat/examples/tutorials", I got error message below: [d3m956 at olympus tutorials]$ ./ex12 -f0 myA.dat [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Read from file failed! [0]PETSC ERROR: Read past end of file! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex12 on a arch-linu named olympus.local by d3m956 Tue Jun 26 13:51:28 2012 [0]PETSC ERROR: Libraries linked from /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscBinaryRead() line 271 in src/sys/fileio/sysio.c [0]PETSC ERROR: PetscBinarySynchronizedRead() line 558 in src/sys/fileio/sysio.c [0]PETSC ERROR: PetscViewerBinaryRead() line 764 in src/sys/viewer/impls/binary/binv.c [0]PETSC ERROR: PetscViewerBinaryReadVecHeader_Private() line 24 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad_Binary() line 97 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad_Default() line 348 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad() line 1111 in src/vec/vec/interface/vector.c [0]PETSC ERROR: main() line 92 in src/mat/examples/tutorials/ex12.c It said "Read from file failed! Read past end of file!" Anything wrong? I was using the example code without changing anything. Please help me to figure out the problem. Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 26 17:07:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 14:07:57 -0800 Subject: [petsc-users] potential problem in dageometry In-Reply-To: <4FEA1B8C.2040105@txcorp.com> References: <4FEA1B8C.2040105@txcorp.com> Message-ID: Matt was a bad boy and forgot to remove the debugging print statements and check that it does the correct thing with complex before pushing. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/e2c8cd4c372b On Tue, Jun 26, 2012 at 12:29 PM, Paul Mullowney wrote: > Hi All, > > I'm trying to build PETSc with the configuration: > ./configure -with-debugging=no --with-mpi=1 --with-mpi-dir=/usr/mpi/gcc/**openmpi-1.4.3 > --CFLAGS="-g -O3" --with-precision=double --with-clanguage=C++ > --with-scalar-type=complex --with-thrust=0 --with-cusp=0 --with-cuda=0 > > I see the errors listed below. I'm wondering if the code in > DMDAComputeCellGeometry_2D function should be using PetscReal instead of > PetscScalar? > > Thanks, > -Paul > > > dageometry.c: In function ?PetscErrorCode DMDAComputeCellGeometry_2D(_p_**DM*, > const PetscScalar*, const PetscReal*, PetscReal*, PetscReal*, > PetscScalar*)?: > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:323: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:324: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:324: warning: cannot pass objects of non-POD type ?const > struct PetscScalar? through ?...?; call will abort at runtime > dageometry.c:325: error: cannot convert ?std::complex? to ?double? > in assignment > dageometry.c:325: error: cannot convert ?std::complex? to ?double? > in assignment > dageometry.c:326: error: cannot convert ?std::complex? to ?double? > in assignment > dageometry.c:326: error: cannot convert ?std::complex? to ?double? > in assignment > dageometry.c:328: error: cannot convert ?std::complex? to > ?PetscReal? in assignment > dageometry.c: In function ?PetscErrorCode DMDAComputeCellGeometry(_p_DM***, > PetscInt, PetscQuadrature*, PetscReal*, PetscReal*, PetscReal*, > PetscReal*)?: > dageometry.c:352: error: cannot convert ?const std::complex? to > ?double? in assignment > dageometry.c:357: error: cannot convert ?PetscReal*? to ?PetscScalar*? for > argument ?6? to ?PetscErrorCode DMDAComputeCellGeometry_2D(_p_**DM*, > const PetscScalar*, const PetscReal*, PetscReal*, PetscReal*, PetscScalar*)? > /usr/bin/ar: dageometry.o: No such file or directory > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Tue Jun 26 17:31:28 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Tue, 26 Jun 2012 18:31:28 -0400 Subject: [petsc-users] Scaling GMRES residual Message-ID: <20120626183128.c88up6ytgk8gowso@cubmail.cc.columbia.edu> Hello, I would like to use GMRES for a monolithic solution of a multiphysics problem. There are drastically different units in the problem and so the residual norm is dominated by the terms with the largest units. Is there a way to tell petsc to scale the residual before taking the norm to avoid this? Thanks Colin -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From jedbrown at mcs.anl.gov Tue Jun 26 17:33:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 14:33:58 -0800 Subject: [petsc-users] Scaling GMRES residual In-Reply-To: <20120626183128.c88up6ytgk8gowso@cubmail.cc.columbia.edu> References: <20120626183128.c88up6ytgk8gowso@cubmail.cc.columbia.edu> Message-ID: On Tue, Jun 26, 2012 at 2:31 PM, Colin McAuliffe wrote: > Hello, > > I would like to use GMRES for a monolithic solution of a multiphysics > problem. There are drastically different units in the problem and so the > residual norm is dominated by the terms with the largest units. Is there a > way to tell petsc to scale the residual before taking the norm to avoid > this? > You can "fix" it with preconditioning, but then you can only use the preconditioned residual. You should really fix the units when formulating the problem. (This does not require formal nondimensionalization, just introduced units so that state and residuals are of order 1.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 26 18:01:46 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 15:01:46 -0800 Subject: [petsc-users] Loading Binary file problem In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018F1929B8E0@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018F1929B8E0@EMAIL04.pnl.gov> Message-ID: On Tue, Jun 26, 2012 at 1:09 PM, Jin, Shuangshuang < Shuangshuang.Jin at pnnl.gov> wrote: > Hello everyone, > > I created a 4*3 matrix A in matlab, saved it to binary file ?myA.dat? > using PetscBinaryWrite(?myA.dat?,sparse(A)). Everything seems fine to me > because I can read the binary file ?myA.dat? by PetscBinaryRead('myA.dat' > )correctly. > > A = [1 2 0; > 2 0 5; > 0 3 0; > 1 4 1] > PetscBinaryWrite('myA.dat',sparse(A)) > PetscBinaryRead('myA.dat') > > However, when I tried to load the binary file ?myA.dat? in PETSc using the > example code ?ex12.c? under ?petsc-3.3-p0/src/mat/examples/tutorials?, I > got error message below: > > [d3m956 at olympus tutorials]$ ./ex12 -f0 myA.dat > This program expects a matrix followed by a vector in the file. You can use -rhs 0 to skip reading the right hand side vector from the file, it will use a vector of all ones. > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Read from file failed! > [0]PETSC ERROR: Read past end of file! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 > CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex12 on a arch-linu named olympus.local by d3m956 Tue > Jun 26 13:51:28 2012 > [0]PETSC ERROR: Libraries linked from > /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran > --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PetscBinaryRead() line 271 in src/sys/fileio/sysio.c > [0]PETSC ERROR: PetscBinarySynchronizedRead() line 558 in > src/sys/fileio/sysio.c > [0]PETSC ERROR: PetscViewerBinaryRead() line 764 in > src/sys/viewer/impls/binary/binv.c > [0]PETSC ERROR: PetscViewerBinaryReadVecHeader_Private() line 24 in > src/vec/vec/utils/vecio.c > [0]PETSC ERROR: VecLoad_Binary() line 97 in src/vec/vec/utils/vecio.c > [0]PETSC ERROR: VecLoad_Default() line 348 in src/vec/vec/utils/vecio.c > [0]PETSC ERROR: VecLoad() line 1111 in src/vec/vec/interface/vector.c > [0]PETSC ERROR: main() line 92 in src/mat/examples/tutorials/ex12.c > > It said ?Read from file failed! Read past end of file!? Anything wrong? I > was using the example code without changing anything. Please help me to > figure out the problem. > > Thanks, > Shuangshuang > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Shuangshuang.Jin at pnnl.gov Tue Jun 26 18:13:50 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 26 Jun 2012 16:13:50 -0700 Subject: [petsc-users] Rectangular matrix Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018F1929B978@EMAIL04.pnl.gov> Hello, I would like to ask if there's any special setup for PTESc to solve an Ax=b problem where A is a rectangular matrix instead of a square matrix. For example, when my A matrix is 3*3, the problem can be solved properly. But when my A matrix is 4*3, the error message is: [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Incompatible vector local lengths 4 != 3! PS: I created the matrix in SeqAIJ format and I could view the matrix was constructed correctly by MatView. // determine number of nonzeros per row in the new matrix PetscMalloc((n+1)*sizeof(PetscInt),&cnt); for (i=0;i From Shuangshuang.Jin at pnnl.gov Tue Jun 26 18:31:27 2012 From: Shuangshuang.Jin at pnnl.gov (Jin, Shuangshuang) Date: Tue, 26 Jun 2012 16:31:27 -0700 Subject: [petsc-users] Loading Binary file problem In-Reply-To: References: <6778DE83AB681D49BFC2CD850610FEB1018F1929B8E0@EMAIL04.pnl.gov> Message-ID: <6778DE83AB681D49BFC2CD850610FEB1018F1929B98C@EMAIL04.pnl.gov> Jed, thanks for pointing out that the program expects a matrix followed by a vector in the file. I created a new binary file ?myAb.dat? in matlab by adding the b vector. A = [1 2 0; 2 0 5; 0 3 0; 1 4 1] b = [1 2 3]' PetscBinaryWrite('myAb.dat',sparse(A),b) [AA bb] = PetscBinaryRead('myAb.dat') The PETSc program works with this input file now: $ ./ex12 -f0 myAb.dat Matrix Object: 1 MPI processes type: seqaij row 0: (0, 1) (1, 2) row 1: (0, 2) (2, 5) row 2: (1, 3) row 3: (0, 1) (1, 4) (2, 1) Matrix Object: 1 MPI processes type: seqaij row 0: (0, 1) (1, 2) (4, 1) row 1: (0, 2) (2, 5) (4, 2) row 2: (1, 3) (4, 3) row 3: (0, 1) (1, 4) (2, 1) (4, 2.22295e-319) row 4: (0, 1) (1, 2) (2, 3) (3, 2.22295e-319) (4, 3) Thanks, Shuangshuang From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, June 26, 2012 4:02 PM To: PETSc users list Subject: Re: [petsc-users] Loading Binary file problem On Tue, Jun 26, 2012 at 1:09 PM, Jin, Shuangshuang > wrote: Hello everyone, I created a 4*3 matrix A in matlab, saved it to binary file ?myA.dat? using PetscBinaryWrite(?myA.dat?,sparse(A)). Everything seems fine to me because I can read the binary file ?myA.dat? by PetscBinaryRead('myA.dat')correctly. A = [1 2 0; 2 0 5; 0 3 0; 1 4 1] PetscBinaryWrite('myA.dat',sparse(A)) PetscBinaryRead('myA.dat') However, when I tried to load the binary file ?myA.dat? in PETSc using the example code ?ex12.c? under ?petsc-3.3-p0/src/mat/examples/tutorials?, I got error message below: [d3m956 at olympus tutorials]$ ./ex12 -f0 myA.dat This program expects a matrix followed by a vector in the file. You can use -rhs 0 to skip reading the right hand side vector from the file, it will use a vector of all ones. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Read from file failed! [0]PETSC ERROR: Read past end of file! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex12 on a arch-linu named olympus.local by d3m956 Tue Jun 26 13:51:28 2012 [0]PETSC ERROR: Libraries linked from /pic/projects/mca/ss/PETSC/petsc-3.3-p0/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Thu Jun 14 17:00:19 2012 [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscBinaryRead() line 271 in src/sys/fileio/sysio.c [0]PETSC ERROR: PetscBinarySynchronizedRead() line 558 in src/sys/fileio/sysio.c [0]PETSC ERROR: PetscViewerBinaryRead() line 764 in src/sys/viewer/impls/binary/binv.c [0]PETSC ERROR: PetscViewerBinaryReadVecHeader_Private() line 24 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad_Binary() line 97 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad_Default() line 348 in src/vec/vec/utils/vecio.c [0]PETSC ERROR: VecLoad() line 1111 in src/vec/vec/interface/vector.c [0]PETSC ERROR: main() line 92 in src/mat/examples/tutorials/ex12.c It said ?Read from file failed! Read past end of file!? Anything wrong? I was using the example code without changing anything. Please help me to figure out the problem. Thanks, Shuangshuang -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Jun 26 18:43:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 26 Jun 2012 15:43:19 -0800 Subject: [petsc-users] Rectangular matrix In-Reply-To: <6778DE83AB681D49BFC2CD850610FEB1018F1929B978@EMAIL04.pnl.gov> References: <6778DE83AB681D49BFC2CD850610FEB1018F1929B978@EMAIL04.pnl.gov> Message-ID: What do you mean by "solve"? You may be able to use LSQR. On Tue, Jun 26, 2012 at 3:13 PM, Jin, Shuangshuang < Shuangshuang.Jin at pnnl.gov> wrote: > Hello, > > I would like to ask if there?s any special setup for PTESc to > solve an Ax=b problem where A is a rectangular matrix instead of a square > matrix. For example, when my A matrix is 3*3, the problem can be solved > properly. But when my A matrix is 4*3, the error message is: > > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Incompatible vector local lengths 4 != 3! > > PS: I created the matrix in SeqAIJ format and I could view the matrix was > constructed correctly by MatView. > > // determine number of nonzeros per row in the new matrix > PetscMalloc((n+1)*sizeof(PetscInt),&cnt); > for (i=0;i cnt[i]=irow[i+1]-irow[i]; > } > MatCreateSeqAIJ(PETSC_COMM_SELF,n,m,0,cnt,&A); > MatSetOption(A,MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE); > // copy over the matrix entries from the inputs > for (i=0;i PetscMalloc(cnt[i]*sizeof(PetscInt),&col); > for (j=0;j col[j]=icol[irow[i]+j-1]-1; > } > MatSetValues(A,1,&i,cnt[i],col,&ival[irow[i]-1],INSERT_VALUES); > } > MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); > ierr=PetscPrintf(PETSC_COMM_WORLD,"Matrix A:\n");CHKERRQ(ierr); > MatView(A,PETSC_VIEWER_STDOUT_SELF); > > I?m new to PETSc. Sorry if my questions is na?ve to you. But any > comment is highly appreciated! > > Thanks, > Shuangshuang > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paeanball at gmail.com Wed Jun 27 02:59:57 2012 From: paeanball at gmail.com (Bao Kai) Date: Wed, 27 Jun 2012 10:59:57 +0300 Subject: [petsc-users] MatFDColorCreate takes really big portion of the total time Message-ID: Hi, Barry, I changed the extension of the file and clean a little bit to attach it. It is a parallel code. Best Regards, Kai Parallel or sequential? Barry Please send entire -log_summary output On Jun 26, 2012, at 2:22 PM, Bao Kai wrote: > Hi, all, > > I use the SNES in petsc-3.2 to solve my problem. The problem is a 3-D > finite difference problem with structured grid. > > I use MatFDColorCreate to generate the Jacobian matrix. I just found > that when the size of problem is big, MatFDColorCreate takes really > long time. The following results is the summary with size of the mesh > to be 1000^3. 90% of the time is costed in MatFDColorCreate. > > 237 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 > 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 238 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 > 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > 239 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 > 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 > 240 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 > 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 > 241 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 > 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 > > And the following the code I use. > > 262 call DMGetMatrix(solv%da, MATAIJ, solv%jac,ierr) > 263 call > DMGetColoring(solv%da,IS_COLORING_GLOBAL,MATAIJ,iscoloring,ierr) > 264 call MatFDColoringCreate(solv%jac,iscoloring,matfdcoloring,ierr) > 265 call > MatFDColoringSetFunction(matfdcoloring,FormFunction,equ,ierr) > 266 call MatFDColoringSetFromOptions(matfdcoloring,ierr) > 267 call SNESSetJacobian(solv%snes, solv%jac, > solv%jac,SNESDefaultComputeJacobianColor, matfdcoloring, ierr) > 268 call ISColoringDestroy(iscoloring,ierr) > > I am wondering if there is anything I can do to improve this problem. > > Thank you very much. > > Best Reg -------------- next part -------------- RST runs on 32768 processors. Start the simulation of RST_advection on a 3D rectanguluar mesh of size 1000x1000x1000. ................. ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./rst on a arch-shah named ionode101 with 32768 processors, by Unknown Mon May 28 14:40:44 2012 Using Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54 CDT 2011 Max Max/Min Avg Total Time (sec): 3.632e+03 1.00000 3.632e+03 Objects: 2.450e+02 1.00000 2.450e+02 Flops: 3.878e+10 1.11767 3.606e+10 1.182e+15 Flops/sec: 1.068e+07 1.11767 9.927e+06 3.253e+11 MPI Messages: 2.843e+05 3.71427 2.662e+05 8.723e+09 MPI Message Lengths: 5.701e+08 2.19550 1.977e+03 1.725e+13 MPI Reductions: 2.161e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 3.6323e+03 100.0% 1.1816e+15 100.0% 8.723e+09 100.0% 1.977e+03 100.0% 2.161e+04 100.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage PetscBarrier 10 1.0 3.0198e-02 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESSolve 10 1.0 3.4011e+02 1.0 3.88e+10 1.1 8.7e+09 2.0e+03 2.1e+04 9100100100 96 9100100100 96 3474169 SNESLineSearch 20 1.0 7.0355e-01 1.0 4.06e+07 1.1 3.2e+07 2.0e+03 8.0e+01 0 0 0 0 0 0 0 0 0 0 1759413 SNESFunctionEval 30 1.0 4.8486e-01 1.2 0.00e+00 0.0 2.4e+07 2.0e+03 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESJacobianEval 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 VecDot 10272 1.0 2.7129e+01 7.8 6.73e+08 1.1 0.0e+00 0.0e+00 1.0e+04 0 2 0 0 48 0 2 0 0 48 757257 VecDotNorm2 5126 1.0 2.4923e+01 8.2 6.72e+08 1.1 0.0e+00 0.0e+00 5.1e+03 0 2 0 0 24 0 2 0 0 24 822681 VecNorm 5206 1.0 2.7628e+00 2.3 3.41e+08 1.1 0.0e+00 0.0e+00 5.2e+03 0 1 0 0 24 0 1 0 0 24 3768681 VecCopy 640 1.0 1.1987e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 10332 1.0 8.4211e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 540 1.0 1.5848e-01 1.7 3.54e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6814569 VecAXPBYCZ 10252 1.0 1.0187e+01 1.3 1.34e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 4025498 VecWAXPY 10272 1.0 2.6825e+00 1.2 6.73e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 7651013 VecScatterBegin 10892 1.0 4.1604e+00 2.1 0.00e+00 0.0 8.7e+09 2.0e+03 0.0e+00 0 0100100 0 0 0100100 0 0 VecScatterEnd 10892 1.0 2.1351e+01 6.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecReduceArith 20 1.0 4.5640e-03 1.2 1.31e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 8764008 VecReduceComm 10 1.0 3.6807e-02 6.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatMult 10272 1.0 1.4871e+02 1.3 1.82e+10 1.1 8.2e+09 2.0e+03 0.0e+00 4 47 94 94 0 4 47 94 94 0 3722481 MatSolve 10272 1.0 1.3996e+02 1.1 1.67e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 43 0 0 0 4 43 0 0 0 3641627 MatLUFactorNum 20 1.0 2.4788e+00 1.1 1.39e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1707587 MatILUFactorSym 1 1.0 2.2120e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 50 1.0 2.6742e+0015.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 50 1.0 8.5123e-01 1.1 0.00e+00 0.0 1.6e+07 5.0e+02 1.1e+02 0 0 0 0 1 0 0 0 0 1 0 MatGetRowIJ 1 1.0 5.0068e-06 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSubMatrice 20 1.0 7.3212e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.3502e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 39 1.0 9.2822e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorCreate 10 1.0 3.2863e+03 1.0 0.00e+00 0.0 1.6e+07 5.0e+02 6.8e+02 90 0 0 0 3 90 0 0 0 3 0 MatFDColorApply 20 1.0 2.5288e+01 1.0 3.54e+07 1.1 4.6e+08 2.0e+03 8.0e+01 1 0 5 5 0 1 0 5 5 0 42708 MatFDColorFunc 560 1.0 9.5386e+00 1.3 0.00e+00 0.0 4.5e+08 2.0e+03 0.0e+00 0 0 5 5 0 0 0 5 5 0 0 KSPSetup 40 1.0 1.2743e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 20 1.0 3.1394e+02 1.0 3.87e+10 1.1 8.2e+09 2.0e+03 2.1e+04 9100 94 94 95 9100 94 94 95 3756190 PCSetUp 40 1.0 3.2385e+00 1.1 1.39e+08 1.1 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 1307021 PCSetUpOnBlocks 10292 1.0 1.4383e+02 1.1 1.69e+10 1.1 0.0e+00 0.0e+00 3.0e+00 4 43 0 0 0 4 43 0 0 0 3573042 PCApply 10272 1.0 1.4149e+02 1.1 1.67e+10 1.1 0.0e+00 0.0e+00 0.0e+00 4 43 0 0 0 4 43 0 0 0 3602261 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 812 0 Distributed Mesh 1 1 316976 0 Vector 87 87 14789660 0 Vector Scatter 22 22 13992 0 Index Set 76 76 565644 0 IS L to G Mapping 11 11 160868 0 Matrix 32 32 134526420 0 Matrix FD Coloring 10 10 69912560 0 Viewer 1 0 0 0 Krylov Solver 2 2 1436 0 Preconditioner 2 2 1100 0 ======================================================================================================================== Average time to get PetscTime(): 3.38554e-06 Average time for MPI_Barrier(): 4.19617e-06 Average time for zero size MPI_Send(): 1.40555e-05 #PETSc Option Table entries: -log_summary -snes_monitor #End of PETSc Option Table entries Compiled with FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 4 sizeof(PetscScalar) 8 Configure run at: Sat Nov 26 18:15:55 2011 Configure options: --known-bits-per-byte=8 --known-level1-dcache-assoc=0 --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768 --known-memcmp-ok --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=4 --known-sizeof-short=2 --known-sizeof-size_t=4 --known-sizeof-void-p=4 --with-batch=1 --with-blas-lapack-lib=" -L/home/aron/ksl/lapack/pdc/ppc450d/lib -L/home/aron/ksl/cblas/pdc/ppc450d/lib -L/bgsys/ibm_essl/sles10/prod/opt/ibmmath/lib -L/opt/ibmcmp/xlsmp/bg/1.7/lib -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/xlf/bg/11.1/bglib -llapack -lcblas -lesslbg -lxlf90_r -lxlopt -lxlsmp -lxl -lxlfmath" --with-cc=mpixlc_r --with-cxx=mpixlcxx_r --with-debugging=0 --with-fc="mpixlf77_r -qnosave" --with-fortran-kernels=1 --with-is-color-value-type=short --with-shared-libraries=0 --with-x=0 -COPTFLAGS="-O3 -qarch=450d -qtune=450 -qmaxmem=-1" -CXXOPTFLAGS="-O3 -qarch=450d -qtune=450 -qmaxmem=-1" -FOPTFLAGS="-O3 -qarch=450d -qtune=450 -qmaxmem=-1" PETSC_ARCH=arch-shaheen ----------------------------------------- Libraries compiled on Sat Nov 26 18:15:55 2011 on fen1 Machine characteristics: Linux-2.6.16.60-0.74.7-ppc64-ppc64-with-SuSE-10-ppc Using PETSc directory: /opt/share/petsc/3.2-p5/bgp Using PETSc arch: arch-shaheen ----------------------------------------- Using C compiler: mpixlc_r -O3 -qarch=450d -qtune=450 -qmaxmem=-1 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: mpixlf77_r -qnosave -O3 -qarch=450d -qtune=450 -qmaxmem=-1 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/opt/share/petsc/3.2-p5/bgp/arch-shaheen/include -I/opt/share/petsc/3.2-p5/bgp/include -I/opt/share/petsc/3.2-p5/bgp/include -I/opt/share/petsc/3.2-p5/bgp/arch-shaheen/include -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/include -I/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/include ----------------------------------------- Using C linker: mpixlc_r Using Fortran linker: mpixlf77_r -qnosave Using libraries: -Wl,-rpath,/opt/share/petsc/3.2-p5/bgp/arch-shaheen/lib -L/opt/share/petsc/3.2-p5/bgp/arch-shaheen/lib -lpetsc -lpthread -L/home/aron/ksl/lapack/pdc/ppc450d/lib -L/home/aron/ksl/cblas/pdc/ppc450d/lib -L/bgsys/ibm_essl/sles10/prod/opt/ibmmath/lib -L/opt/ibmcmp/xlsmp/bg/1.7/lib -L/opt/ibmcmp/xlmass/bg/4.4/bglib -L/opt/ibmcmp/xlf/bg/11.1/bglib -llapack -lcblas -lesslbg -lxlf90_r -lxlopt -lxlsmp -lxl -lxlfmath -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/lib -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/lib -L/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/runtime/SPI -L/opt/ibmcmp/xlsmp/bg/1.7/bglib -L/opt/ibmcmp/vac/bg/9.0/bglib -L/opt/ibmcmp/vacpp/bg/9.0/bglib -L/bgsys/drivers/ppcfloor/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2 -L/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/lib -Wl,-rpath,/opt/ibmcmp/lib/bg/bglib -ldl -Wl,-rpath,/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/default/lib -lmpich.cnk -lopa -Wl,-rpath,/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/comm/sys/lib -ldcmf.cnk -ldcmfcoll.cnk -lpthread -Wl,-rpath,/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/runtime/SPI -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -lxlf90_r -lxlomp_ser -lxlfmath -lm -ldl -lmpich.cnk -lopa -ldcmf.cnk -ldcmfcoll.cnk -lpthread -lSPI.cna -lrt -lxlopt -lxl -lgcc_eh -ldl ----------------------------------------- Finish RST_CO2. Finish interpreting the file infile_RSTi.m by RST. after mpi_finalize: ierr = 0 From cirilloalberto at gmail.com Wed Jun 27 03:58:13 2012 From: cirilloalberto at gmail.com (Alberto Cirillo) Date: Wed, 27 Jun 2012 10:58:13 +0200 Subject: [petsc-users] VECSEQCUSP and fortran Message-ID: Hello, I would know if it's possible to use the VECSEQCUSP type for petsc vectors, developing in Fortran. I've some problems about that. PETSC Version is 3.2-p7 Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 27 05:27:10 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Jun 2012 04:27:10 -0600 Subject: [petsc-users] VECSEQCUSP and fortran In-Reply-To: References: Message-ID: On Wed, Jun 27, 2012 at 2:58 AM, Alberto Cirillo wrote: > Hello, > > I would know if it's possible to use the VECSEQCUSP type for petsc > vectors, developing in Fortran. I've some problems about that. > PETSC Version is 3.2-p7 > Use should use petsc-dev to use GPUs, since the CUDA toolkit changes so fast. Yes, you can use it from Fortran. Matt > Thanks! > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Wed Jun 27 08:55:37 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Wed, 27 Jun 2012 21:55:37 +0800 (CST) Subject: [petsc-users] ill-conditioned problems in PETSc Message-ID: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> Hello, I want to deal with the ill-conditioned problems of Terzaghi's one-dimensional consolidation finite element equations in geotechnical engineering. So is there special support on ill-conditioned problems using ksp in PETSc. Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jun 27 09:46:18 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 27 Jun 2012 08:46:18 -0600 Subject: [petsc-users] ill-conditioned problems in PETSc In-Reply-To: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> References: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> Message-ID: On Wed, Jun 27, 2012 at 7:55 AM, w_ang_temp wrote: > Hello, > > I want to deal with the ill-conditioned problems of Terzaghi's > one-dimensional consolidation > > finite element equations in geotechnical engineering. So is there special > support on ill-conditioned > > problems using ksp in PETSc. > Use a direct solver, like built-in LU, or in parallel, MUMPS (--download-mumps). Matt > Thanks. > > Jim > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Jun 27 13:38:25 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 27 Jun 2012 13:38:25 -0500 Subject: [petsc-users] ill-conditioned problems in PETSc In-Reply-To: References: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> Message-ID: <071D9A22-DF69-488C-ADCE-BCEF3563542E@mcs.anl.gov> Also if it the condition number is, say, larger than 10^10 you likely would benefit from using quad precision for the calculations. http://www.mcs.anl.gov/petsc/documentation/faq.html#precision You will not be able to use the external solvers with those mode but since it is 1d likely running sequentially will be fine. Barry On Jun 27, 2012, at 9:46 AM, Matthew Knepley wrote: > On Wed, Jun 27, 2012 at 7:55 AM, w_ang_temp wrote: > Hello, > > I want to deal with the ill-conditioned problems of Terzaghi's one-dimensional consolidation > > finite element equations in geotechnical engineering. So is there special support on ill-conditioned > > problems using ksp in PETSc. > > > Use a direct solver, like built-in LU, or in parallel, MUMPS (--download-mumps). > > Matt > > Thanks. > > Jim > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From balay at mcs.anl.gov Wed Jun 27 15:49:36 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 27 Jun 2012 15:49:36 -0500 (CDT) Subject: [petsc-users] Search the mailing list does not work In-Reply-To: References: <4FD87A57.5050305@tu-dresden.de> Message-ID: Search is restored now. Satish On Wed, 13 Jun 2012, Jed Brown wrote: > Thanks, we're looking into it. > > On Wed, Jun 13, 2012 at 6:32 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > > > My most important knowledge database does not work anymore :) Searching in > > http://lists.mcs.anl.gov/**pipermail/petsc-users/and > > http://lists.mcs.anl.gov/**pipermail/petsc-devfails. Has anybody changed something on the configuration of the mailing > > list? > > > > Thomas > > > From paulm at txcorp.com Wed Jun 27 17:09:47 2012 From: paulm at txcorp.com (Paul Mullowney) Date: Wed, 27 Jun 2012 16:09:47 -0600 Subject: [petsc-users] question about MatDuplicate_SeqAIJ_Inode Message-ID: <4FEB84AB.5010909@txcorp.com> Hi All, In inode.c, the function MatDuplicate_SeqAIJ_Inode(Mat A,MatDuplicateOption cpvalues,Mat *C) resets the function pointers of A. Shouldn't the functions pointers of C be set instead since the data to C is being modified in this function. -Paul From dharmareddy84 at gmail.com Wed Jun 27 20:06:03 2012 From: dharmareddy84 at gmail.com (Dharmendar Reddy) Date: Wed, 27 Jun 2012 20:06:03 -0500 Subject: [petsc-users] an MPI/Fortran qery not related to PETSc Message-ID: Hello, The question is not related to PETSc, can some one help me with user defined MPI reduce operation in FORTRAN. I have defined a MPI_TYPE_CONTIGUOUS and am able to send and receive it. When i try to to MPI_Reduce, i get error. I have attached the code. Thanks Reddy -- ----------------------------------------------------- Dharmendar Reddy Palle Graduate Student Microelectronics Research center, University of Texas at Austin, 10100 Burnet Road, Bldg. 160 MER 2.608F, TX 78758-4445 e-mail: dharmareddy84 at gmail.com Phone: +1-512-350-9082 United States of America. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MPIDTTest.F90 Type: application/octet-stream Size: 3714 bytes Desc: not available URL: From w_ang_temp at 163.com Thu Jun 28 08:40:32 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Thu, 28 Jun 2012 21:40:32 +0800 (CST) Subject: [petsc-users] ill-conditioned problems in PETSc In-Reply-To: <071D9A22-DF69-488C-ADCE-BCEF3563542E@mcs.anl.gov> References: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> <071D9A22-DF69-488C-ADCE-BCEF3563542E@mcs.anl.gov> Message-ID: <32091ff2.24b58.13833525d35.Coremail.w_ang_temp@163.com> Hello, In fact, I will deal with the large sparse symmetric or unsymmetric (depending on the constitutive model used for the soil and the formulation of equations) linear system which usually occurs in geotechnical problems. As you know, In most geotechnical problems, the coefficient matrix can be severely ill-conditioned, thus calling for the development of robust and efficient preconditioners and iterative methods. The iterative method and parallel solver are my choices, actually are required by my professor. As I know, MUMPS is just an external solver like Matlab. So is it a right way to analyse the preconditioners and iterative methods provided by PETSc like CG, GMRES, Jacobi, sor, etc. for my work? Also I do not quite understand the second part of what Barry said ('You will not be able to use the external solvers with those mode'). So, what does 'those mode' mean. Using quad precision? Thanks a lot. Jim At 2012-06-28 02:38:25,"Barry Smith" wrote: > > Also if it the condition number is, say, larger than 10^10 you likely would benefit from using quad precision for the calculations. http://www.mcs.anl.gov/petsc/documentation/faq.html#precision > > You will not be able to use the external solvers with those mode but since it is 1d likely running sequentially will be fine. > > Barry > > >On Jun 27, 2012, at 9:46 AM, Matthew Knepley wrote: > >> On Wed, Jun 27, 2012 at 7:55 AM, w_ang_temp wrote: >> Hello, >> >> I want to deal with the ill-conditioned problems of Terzaghi's one-dimensional consolidation >> >> finite element equations in geotechnical engineering. So is there special support on ill-conditioned >> >> problems using ksp in PETSc. >> >> >> Use a direct solver, like built-in LU, or in parallel, MUMPS (--download-mumps). >> >> Matt >> >> Thanks. >> >> Jim >> >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.de-soza at edf.fr Thu Jun 28 08:59:21 2012 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Thu, 28 Jun 2012 15:59:21 +0200 Subject: [petsc-users] Avoiding malloc overhead for unstructured finite element meshes Message-ID: Dear PETSc Users, We're experiencing performances issues after having switched to fully distributed meshes in our in-house code and would like your opinion on the matter. In the current version of our structural mechanics FEA software (http://www.code-aster.org/), all MPI processes have knowledge of the whole matrix and therefore can easily pass it to PETSc without the need for any communication. In a nutshell, stash is empty after MatSetValues and no mallocs occur during assembly. We're now building a distributed version of the software with each process reading its own subdomain in order to save memory. The mesh was partitioned with Metis and as a first approach we built a simple partition of the degrees of freedom based on the gradual subdomains. This eliminates the need for Application Ordering but yields an unbalanced decomposition in terms of rows. If we take an example with 2 MPI processes : processor 0 will have more unknowns than processor 1 and will receive entries lying on the interface whereas processor 1 will have all entries locally. PETSc manual states that "It is fine to generate some entries on the "wrong" process. Often this can lead to cleaner, simpler, less buggy codes. One should never make code overly complicated in order to generate all values locally. Rather, one should organize the code in such a way that most values are generated locally." Judging from the performance we obtain on a simple cube with two processes, it seems we have generated too much entries on the wrong process. Indeed our distributed code runs slower than the current one. However the stash does not seem to contain that much (650 000 over a total of 50 000 000 nnz). We have attached the output obtained with "-info" as well as the "-log_summary" profiling. Most of the time is spent in the assembly and a lot of mallocs occur. What's your advice on this ? Is working with ghost cells the only option ? We were wondering if we could preallocate the stash for example to decrease the number of mallocs. Regards, Thomas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_profiling.log Type: application/octet-stream Size: 10146 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_info.log Type: application/octet-stream Size: 4668 bytes Desc: not available URL: From stali at geology.wisc.edu Thu Jun 28 10:16:59 2012 From: stali at geology.wisc.edu (Tabrez Ali) Date: Thu, 28 Jun 2012 10:16:59 -0500 Subject: [petsc-users] Avoiding malloc overhead for unstructured finite element meshes In-Reply-To: References: Message-ID: <4FEC756B.3060403@geology.wisc.edu> Did you renumber your on-rank nodes after partitioning? T On 06/28/2012 08:59 AM, Thomas DE-SOZA wrote: > > Dear PETSc Users, > > We're experiencing performances issues after having switched to fully > distributed meshes in our in-house code and would like your opinion on > the matter. > > In the current version of our structural mechanics FEA software > (http://www.code-aster.org/), all MPI processes have knowledge of the > whole matrix and therefore can easily pass it to PETSc without the > need for any communication. In a nutshell, stash is empty after > MatSetValues and no mallocs occur during assembly. > We're now building a distributed version of the software with each > process reading its own subdomain in order to save memory. The mesh > was partitioned with Metis and as a first approach we built a simple > partition of the degrees of freedom based on the gradual subdomains. > This eliminates the need for Application Ordering but yields an > unbalanced decomposition in terms of rows. If we take an example with > 2 MPI processes : processor 0 will have more unknowns than processor 1 > and will receive entries lying on the interface whereas processor 1 > will have all entries locally. > > PETSc manual states that "It is fine to generate some entries on the > "wrong" process. Often this can lead to cleaner, simpler, less buggy > codes. One should never make code overly complicated in order to > generate all values locally. Rather, one should organize the code in > such a way that most values are generated locally." > Judging from the performance we obtain on a simple cube with two > processes, it seems we have generated too much entries on the wrong > process. Indeed our distributed code runs slower than the current one. > However the stash does not seem to contain that much (650 000 over a > total of 50 000 000 nnz). We have attached the output obtained with > "-info" as well as the "-log_summary" profiling. Most of the time is > spent in the assembly and a lot of mallocs occur. > > > > > What's your advice on this ? Is working with ghost cells the only > option ? > We were wondering if we could preallocate the stash for example to > decrease the number of mallocs. > > Regards, > > Thomas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations > qui y figurent sont strictement confidentielles. Toute utilisation de > ce Message non conforme ? sa destination, toute diffusion ou toute > publication totale ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit > de le copier, de le faire suivre, de le divulguer ou d'en utiliser > tout ou partie. Si vous avez re?u ce Message par erreur, merci de le > supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en > garder aucune trace sur quelque support que ce soit. Nous vous > remercions ?galement d'en avertir imm?diatement l'exp?diteur par > retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de > toute erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely > for the addressees. The information contained in this Message is > confidential. Any use of information contained in this Message not in > accord with its purpose, any dissemination or disclosure, either whole > or partial, is prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or > use any part of it. If you have received this message in error, please > delete it and all copies from your system and notify the sender > immediately by return message. > > E-mail communication cannot be guaranteed to be timely secure, error > or virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jun 28 10:01:44 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Thu, 28 Jun 2012 10:01:44 -0500 Subject: [petsc-users] ill-conditioned problems in PETSc In-Reply-To: <32091ff2.24b58.13833525d35.Coremail.w_ang_temp@163.com> References: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> <071D9A22-DF69-488C-ADCE-BCEF3563542E@mcs.anl.gov> <32091ff2.24b58.13833525d35.Coremail.w_ang_temp@163.com> Message-ID: Jim, > > In fact, I will deal with the large sparse symmetric or unsymmetric > (depending on the constitutive > model used for the soil and the formulation of equations) linear system > which usually occurs in geotechnical > problems. As you know, In most geotechnical problems, the coefficient > matrix can be severely > ill-conditioned, thus calling for the development of robust and efficient > preconditioners and iterative methods. > > The iterative method and parallel solver are my choices, actually are > required by my professor. As I know, > MUMPS is just an external solver like Matlab. So is it a right way to > analyse the preconditioners and iterative > MUMPS and Superlu_dist are parallel sparse direct solvers for ill-conditioned problems. They can solve problems with 10k unknowns sequentially and scale up to 10+ or 100+ processors, i.e. solve problems with 100k unknowns. Give them a try. Then, explore other preconditioners. methods provided by PETSc like CG, GMRES, Jacobi, sor, etc. for my work? > > Also I do not quite understand the second part of what Barry said > ('You will not be able to use the external > solvers with those mode'). So, what does 'those mode' mean. Using quad > precision? > Yes, high precisions. Start from small problems with robust solvers, then slowly move to large, complex problems. Hong > > > > > At 2012-06-28 02:38:25,"Barry Smith" wrote: > > > > Also if it the condition number is, say, larger than 10^10 you likely would benefit from using quad precision for the calculations. http://www.mcs.anl.gov/petsc/documentation/faq.html#precision > > > > You will not be able to use the external solvers with those mode but since it is 1d likely running sequentially will be fine. > > > > Barry > > > > > >On Jun 27, 2012, at 9:46 AM, Matthew Knepley wrote: > > > >> On Wed, Jun 27, 2012 at 7:55 AM, w_ang_temp wrote: > >> Hello, > >> > >> I want to deal with the ill-conditioned problems of Terzaghi's one-dimensional consolidation > >> > >> finite element equations in geotechnical engineering. So is there special support on ill-conditioned > >> > >> problems using ksp in PETSc. > >> > >> > >> Use a direct solver, like built-in LU, or in parallel, MUMPS (--download-mumps). > >> > >> Matt > >> > >> Thanks. > >> > >> Jim > >> > >> > >> > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> -- Norbert Wiener > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 28 10:36:44 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 28 Jun 2012 07:36:44 -0800 Subject: [petsc-users] Avoiding malloc overhead for unstructured finite element meshes In-Reply-To: References: Message-ID: >From petsc_info.log: 0: [0] MatStashScatterBegin_Private(): No of messages: 0 0: [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. 1: [1] MatAssemblyBegin_MPIAIJ(): Stash has 645696 entries, uses 6 mallocs. This means that rank 1 generates a bunch of entries in rows owned by rank 0, but not vice-versa. The number of entries is somewhat high, but not unreasonable. 1: [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 334296 X 334296; storage space: 0 unneeded,25888950 used 1: [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 1: [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 1: [1] Mat_CheckInode(): Found 111432 nodes of 334296. Limit used: 5. Using Inode routines Rank 1 preallocated correctly, no problem here. 0: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 346647 X 346647; storage space: 9900 unneeded,26861247 used 0: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1608 This number of mallocs is the real problem, you have not preallocated correctly for the "diagonal" block of the matrix on rank 0. Fix preallocation and it will be fast. Everything below is fine. 0: [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 0: [0] Mat_CheckInode(): Found 115549 nodes of 346647. Limit used: 5. Using Inode routines 0: [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter 1: [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 1: [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] VecScatterCreateCommon_PtoS(): Using blocksize 3 scatter 0: [0] VecScatterCreate(): Special case: blocked indices to stride 0: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 346647 X 12234; storage space: 0 unneeded,308736 used 0: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 0: [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 51 1: [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 334296 X 12210; storage space: 0 unneeded,308736 used 1: [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 1: [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 45 On Thu, Jun 28, 2012 at 5:59 AM, Thomas DE-SOZA wrote: > > Dear PETSc Users, > > We're experiencing performances issues after having switched to fully > distributed meshes in our in-house code and would like your opinion on the > matter. > > In the current version of our structural mechanics FEA software ( > http://www.code-aster.org/), all MPI processes have knowledge of the > whole matrix and therefore can easily pass it to PETSc without the need for > any communication. In a nutshell, stash is empty after MatSetValues and no > mallocs occur during assembly. > We're now building a distributed version of the software with each process > reading its own subdomain in order to save memory. The mesh was partitioned > with Metis and as a first approach we built a simple partition of the > degrees of freedom based on the gradual subdomains. This eliminates the > need for Application Ordering but yields an unbalanced decomposition in > terms of rows. If we take an example with 2 MPI processes : processor 0 > will have more unknowns than processor 1 and will receive entries lying on > the interface whereas processor 1 will have all entries locally. > > PETSc manual states that "It is fine to generate some entries on the > "wrong" process. Often this can lead to cleaner, simpler, less buggy codes. > One should never make code overly complicated in order to generate all > values locally. Rather, one should organize the code in such a way that > most values are generated locally." > Judging from the performance we obtain on a simple cube with two > processes, it seems we have generated too much entries on the wrong > process. Indeed our distributed code runs slower than the current one. > However the stash does not seem to contain that much (650 000 over a total > of 50 000 000 nnz). We have attached the output obtained with "-info" as > well as the "-log_summary" profiling. Most of the time is spent in the > assembly and a lot of mallocs occur. > > > > > What's your advice on this ? Is working with ghost cells the only option ? > We were wondering if we could preallocate the stash for example to > decrease the number of mallocs. > > Regards, > > Thomas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Thu Jun 28 10:44:13 2012 From: popov at uni-mainz.de (Anton Popov) Date: Thu, 28 Jun 2012 17:44:13 +0200 Subject: [petsc-users] fieldsplit recursive Message-ID: <4FEC7BCD.7030803@uni-mainz.de> Dear petsc team, I'm trying to use fieldsplit preconditioner for the velocity block in the Stokes system which is also preconditioned by fieldsplit (kind of recursive). Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as follows: mpiexec -np 4 ./ex42 \ -stokes_ksp_type gcr \ -stokes_ksp_rtol 1.0e-6 \ -stokes_pc_type fieldsplit \ -stokes_pc_fieldsplit_type SCHUR \ -stokes_pc_fieldsplit_schur_factorization_type UPPER \ -stokes_fieldsplit_u_ksp_type fgmres \ -stokes_fieldsplit_u_ksp_rtol 1e-3 \ -stokes_fieldsplit_u_pc_type fieldsplit \ -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ -stokes_fieldsplit_u_fieldsplit_pc_type ml \ -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ -stokes_fieldsplit_p_ksp_type preonly \ -stokes_fieldsplit_p_pc_type jacobi \ -stokes_ksp_monitor_blocks \ -mx 16 \ -model 3 gives nicely looking output. But! Repeating the same exercise with petsc-3.3, like this (actually, there is only one difference: factorization -> fact): mpiexec -np 4 ./ex42 \ -stokes_ksp_type gcr \ -stokes_ksp_rtol 1.0e-6 \ -stokes_pc_type fieldsplit \ -stokes_pc_fieldsplit_type SCHUR \ -stokes-pc_fieldsplit_schur_fact_type UPPER \ -stokes_fieldsplit_u_ksp_type fgmres \ -stokes_fieldsplit_u_ksp_rtol 1e-3 \ -stokes_fieldsplit_u_pc_type fieldsplit \ -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ -stokes_fieldsplit_u_fieldsplit_pc_type ml \ -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ -stokes_fieldsplit_p_ksp_type preonly \ -stokes_fieldsplit_p_pc_type jacobi \ -stokes_ksp_monitor_blocks \ -mx 16 \ -model 3 curses me hardly by claiming: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Object is in wrong state! [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 3! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by anton Thu Jun 28 17:06:53 2012 [0]PETSC ERROR: Libraries linked from /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCApply_FieldSplit() line 726 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPFGMRESCycle() line 169 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c [0]PETSC ERROR: KSPSolve_GCR() line 117 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in src/ksp/ksp/examples/tutorials/ex42.c [0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 Similar error appeared in our code after upgrading to petsc-3.3, and we're using similar functionality and options as I posted above. Please explain this issue. An advice how to get rid of the error is also appreciated. Thanks a lot Anton From thomas.de-soza at edf.fr Thu Jun 28 12:29:03 2012 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Thu, 28 Jun 2012 19:29:03 +0200 Subject: [petsc-users] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A__Avoiding_malloc_?= =?iso-8859-1?q?overhead_for_unstructured_finite_element_meshes?= Message-ID: 0: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 346647 X 346647; storage space: 9900 unneeded,26861247 used 0: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1608 This number of mallocs is the real problem, you have not preallocated correctly for the "diagonal" block of the matrix on rank 0. Fix preallocation and it will be fast. Everything below is fine. Since 9900 coefficients were uneeded, we had first thought that enough room was preallocated. >From what you're telling us, I understand that we may have given an overall size which is large enough to contain the diagonal block but whose nnz line by line is not correct hence the mallocs. Is that correct ? Or does it mean that in the preallocation we have to take care of the values that come from the stash of another processor even if they are added to preexisting entries on the process ? Thomas jedbrown at mcs.anl.gov Envoy? par : petsc-users-bounces at mcs.anl.gov 28/06/2012 17:36 Veuillez r?pondre ? petsc-users Pour : petsc-users at mcs.anl.gov cc : nicolas.sellenet at edf.fr, (ccc : Thomas DE-SOZA/A/EDF/FR) Objet : Re: [petsc-users] Avoiding malloc overhead for unstructured finite element meshes >From petsc_info.log: 0: [0] MatStashScatterBegin_Private(): No of messages: 0 0: [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. 1: [1] MatAssemblyBegin_MPIAIJ(): Stash has 645696 entries, uses 6 mallocs. This means that rank 1 generates a bunch of entries in rows owned by rank 0, but not vice-versa. The number of entries is somewhat high, but not unreasonable. 1: [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 334296 X 334296; storage space: 0 unneeded,25888950 used 1: [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 1: [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 1: [1] Mat_CheckInode(): Found 111432 nodes of 334296. Limit used: 5. Using Inode routines Rank 1 preallocated correctly, no problem here. 0: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 346647 X 346647; storage space: 9900 unneeded,26861247 used 0: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 1608 This number of mallocs is the real problem, you have not preallocated correctly for the "diagonal" block of the matrix on rank 0. Fix preallocation and it will be fast. Everything below is fine. 0: [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81 0: [0] Mat_CheckInode(): Found 115549 nodes of 346647. Limit used: 5. Using Inode routines 0: [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter 1: [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 1: [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783 0: [0] VecScatterCreateCommon_PtoS(): Using blocksize 3 scatter 0: [0] VecScatterCreate(): Special case: blocked indices to stride 0: [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 346647 X 12234; storage space: 0 unneeded,308736 used 0: [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 0: [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 51 1: [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 334296 X 12210; storage space: 0 unneeded,308736 used 1: [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 1: [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 45 On Thu, Jun 28, 2012 at 5:59 AM, Thomas DE-SOZA wrote: Dear PETSc Users, We're experiencing performances issues after having switched to fully distributed meshes in our in-house code and would like your opinion on the matter. In the current version of our structural mechanics FEA software (http://www.code-aster.org/), all MPI processes have knowledge of the whole matrix and therefore can easily pass it to PETSc without the need for any communication. In a nutshell, stash is empty after MatSetValues and no mallocs occur during assembly. We're now building a distributed version of the software with each process reading its own subdomain in order to save memory. The mesh was partitioned with Metis and as a first approach we built a simple partition of the degrees of freedom based on the gradual subdomains. This eliminates the need for Application Ordering but yields an unbalanced decomposition in terms of rows. If we take an example with 2 MPI processes : processor 0 will have more unknowns than processor 1 and will receive entries lying on the interface whereas processor 1 will have all entries locally. PETSc manual states that "It is fine to generate some entries on the "wrong" process. Often this can lead to cleaner, simpler, less buggy codes. One should never make code overly complicated in order to generate all values locally. Rather, one should organize the code in such a way that most values are generated locally." Judging from the performance we obtain on a simple cube with two processes, it seems we have generated too much entries on the wrong process. Indeed our distributed code runs slower than the current one. However the stash does not seem to contain that much (650 000 over a total of 50 000 000 nnz). We have attached the output obtained with "-info" as well as the "-log_summary" profiling. Most of the time is spent in the assembly and a lot of mallocs occur. What's your advice on this ? Is working with ghost cells the only option ? We were wondering if we could preallocate the stash for example to decrease the number of mallocs. Regards, Thomas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Jun 28 12:33:56 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 28 Jun 2012 09:33:56 -0800 Subject: [petsc-users] =?utf-8?q?R=C3=A9f=2E_=3A_Re=3A_Avoiding_malloc_ove?= =?utf-8?q?rhead_for_unstructured_finite_element_meshes?= In-Reply-To: References: Message-ID: On Thu, Jun 28, 2012 at 9:29 AM, Thomas DE-SOZA wrote: > Since 9900 coefficients were uneeded, we had first thought that enough > room was preallocated. > From what you're telling us, I understand that we may have given an > overall size which is large enough to contain the diagonal block but whose > nnz line by line is not correct hence the mallocs. Is that correct ? > Unlikely, extra space is dynamically allocated in bigger chunks so there is usually some left over when mallocs are needed in MatSetValues. You should be careful to get nnz correct, but that is not the only problem here. > > Or does it mean that in the preallocation we have to take care of the > values that come from the stash of another processor even if they are added > to preexisting entries on the process ? > You only have to allocate the entries once, but there are entries that were not preallocated at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Jun 28 13:07:35 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Jun 2012 13:07:35 -0500 Subject: [petsc-users] question about MatDuplicate_SeqAIJ_Inode In-Reply-To: <4FEB84AB.5010909@txcorp.com> References: <4FEB84AB.5010909@txcorp.com> Message-ID: <3D7FA4FD-2110-4FB2-885C-93EF40099806@mcs.anl.gov> On Jun 27, 2012, at 5:09 PM, Paul Mullowney wrote: > Hi All, > > In inode.c, the function > > MatDuplicate_SeqAIJ_Inode(Mat A,MatDuplicateOption cpvalues,Mat *C) > > resets the function pointers of A. Shouldn't the functions pointers of C be set instead since the data to C is being modified in this function. > > -Paul Yes. I have fixed this in the petsc 3.3 repo and pulled the fixes into the petsc-dev repo. The fix will also be in the next petsc 3.3 patch Barry From bsmith at mcs.anl.gov Thu Jun 28 13:55:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 28 Jun 2012 13:55:07 -0500 Subject: [petsc-users] fieldsplit recursive In-Reply-To: <4FEC7BCD.7030803@uni-mainz.de> References: <4FEC7BCD.7030803@uni-mainz.de> Message-ID: Anton, This came about because we are now being much more pedantic about the blocksizes of PETSc objects and not allowing them to be causally changed when they shouldn't be. You can resolve this problem by editing the file src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function #undef __FUNCT__ #define __FUNCT__ "PCApply_FieldSplit" static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) { PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; PetscErrorCode ierr; PC_FieldSplitLink ilink = jac->head; PetscInt cnt,bs; PetscFunctionBegin; and add the two lines right here x->map->bs = jac->bs; y->map->bs = jac->bs; then run make cmake in that directory. To resolve this permanently we will need to figure out how to insure those inner vectors are created with the correct block size. Are you willing to share your code with petsc-maint at mcs.anl.gov so that we can reproduce the problem and fix it properly for the long run? (The problem is in PETSc not in your code). Barry On Jun 28, 2012, at 10:44 AM, Anton Popov wrote: > Dear petsc team, > > I'm trying to use fieldsplit preconditioner for the velocity block in the Stokes system which is also preconditioned by > fieldsplit (kind of recursive). > > Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as follows: > > mpiexec -np 4 ./ex42 \ > -stokes_ksp_type gcr \ > -stokes_ksp_rtol 1.0e-6 \ > -stokes_pc_type fieldsplit \ > -stokes_pc_fieldsplit_type SCHUR \ > -stokes_pc_fieldsplit_schur_factorization_type UPPER \ > -stokes_fieldsplit_u_ksp_type fgmres \ > -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > -stokes_fieldsplit_u_pc_type fieldsplit \ > -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > -stokes_fieldsplit_p_ksp_type preonly \ > -stokes_fieldsplit_p_pc_type jacobi \ > -stokes_ksp_monitor_blocks \ > -mx 16 \ > -model 3 > > gives nicely looking output. > > But! Repeating the same exercise with petsc-3.3, like this (actually, there is only one difference: factorization -> fact): > > mpiexec -np 4 ./ex42 \ > -stokes_ksp_type gcr \ > -stokes_ksp_rtol 1.0e-6 \ > -stokes_pc_type fieldsplit \ > -stokes_pc_fieldsplit_type SCHUR \ > -stokes-pc_fieldsplit_schur_fact_type UPPER \ > -stokes_fieldsplit_u_ksp_type fgmres \ > -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > -stokes_fieldsplit_u_pc_type fieldsplit \ > -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > -stokes_fieldsplit_p_ksp_type preonly \ > -stokes_fieldsplit_p_pc_type jacobi \ > -stokes_ksp_monitor_blocks \ > -mx 16 \ > -model 3 > > curses me hardly by claiming: > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 3! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by anton Thu Jun 28 17:06:53 2012 > [0]PETSC ERROR: Libraries linked from /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib > [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 > [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1 > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApply_FieldSplit() line 726 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPFGMRESCycle() line 169 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > [0]PETSC ERROR: KSPSolve_GCR() line 117 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in src/ksp/ksp/examples/tutorials/ex42.c > [0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > Similar error appeared in our code after upgrading to petsc-3.3, and we're using similar functionality and options as I posted above. > > Please explain this issue. An advice how to get rid of the error is also appreciated. > > Thanks a lot > > Anton From onetfbao at gmail.com Thu Jun 28 14:10:00 2012 From: onetfbao at gmail.com (Fang-Bao Tian) Date: Thu, 28 Jun 2012 14:10:00 -0500 Subject: [petsc-users] please sign me off Message-ID: thanks -- Dr. Fang-Bao Tian Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, 37212, USA -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.de-soza at edf.fr Fri Jun 29 04:10:42 2012 From: thomas.de-soza at edf.fr (Thomas DE-SOZA) Date: Fri, 29 Jun 2012 11:10:42 +0200 Subject: [petsc-users] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_=09R=E9f=2E_=3A_R?= =?iso-8859-1?q?e=3A_Avoiding_malloc_overhead_for_unstructured_finite_elem?= =?iso-8859-1?q?ent_meshes?= Message-ID: Or does it mean that in the preallocation we have to take care of the values that come from the stash of another processor even if they are added to preexisting entries on the process ? You only have to allocate the entries once, but there are entries that were not preallocated at all. We haven't been able to narrow the problem in our preallocation yet but by preallocating the entries for the incoming stash values (even if it should not be needed as you pointed out) we're now overestimating the nnz and performance is OK (log attached for those curious) : no more mallocs in MatSetValues. Many thanks for your help ! Thomas jedbrown at mcs.anl.gov Envoy? par : petsc-users-bounces at mcs.anl.gov 28/06/2012 19:33 Veuillez r?pondre ? petsc-users Pour : petsc-users at mcs.anl.gov cc : nicolas.sellenet at edf.fr, (ccc : Thomas DE-SOZA/A/EDF/FR) Objet : Re: [petsc-users] R?f. : Re: Avoiding malloc overhead for unstructured finite element meshes On Thu, Jun 28, 2012 at 9:29 AM, Thomas DE-SOZA wrote: Since 9900 coefficients were uneeded, we had first thought that enough room was preallocated. >From what you're telling us, I understand that we may have given an overall size which is large enough to contain the diagonal block but whose nnz line by line is not correct hence the mallocs. Is that correct ? Unlikely, extra space is dynamically allocated in bigger chunks so there is usually some left over when mallocs are needed in MatSetValues. You should be careful to get nnz correct, but that is not the only problem here. Or does it mean that in the preallocation we have to take care of the values that come from the stash of another processor even if they are added to preexisting entries on the process ? You only have to allocate the entries once, but there are entries that were not preallocated at all. Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_profiling_v2.log Type: application/octet-stream Size: 10146 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: petsc_info_v2.log Type: application/octet-stream Size: 4682 bytes Desc: not available URL: From jedbrown at mcs.anl.gov Fri Jun 29 04:17:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 29 Jun 2012 01:17:13 -0800 Subject: [petsc-users] =?utf-8?b?UsOpZi4gOiBSZTogUsOpZi4gOiBSZTogQXZvaWRp?= =?utf-8?q?ng_malloc_overhead_for_unstructured_finite_element_meshe?= =?utf-8?q?s?= In-Reply-To: References: Message-ID: On Fri, Jun 29, 2012 at 1:10 AM, Thomas DE-SOZA wrote: > > Or does it mean that in the preallocation we have to take care of the > values that come from the stash of another processor even if they are added > to preexisting entries on the process ? > > You only have to allocate the entries once, but there are entries that > were not preallocated at all. > > We haven't been able to narrow the problem in our preallocation yet but by > preallocating the entries for the incoming stash values (even if it should > not be needed as you pointed out) we're now overestimating the nnz and > performance is OK (log attached for those curious) : no more mallocs in > MatSetValues. > Run in a debugger with -mat_new_nonzero_allocation_err. It will stop at the first inserted value that was not preallocated. This usually helps you to figure out why that value (or one already inserted in that row) was not preallocated. > > Many thanks for your help ! > > Thomas > > > > > *jedbrown at mcs.anl.gov* > Envoy? par : petsc-users-bounces at mcs.anl.gov > > 28/06/2012 19:33 > Veuillez r?pondre ? petsc-users > > Pour : petsc-users at mcs.anl.gov > cc : nicolas.sellenet at edf.fr, (ccc : Thomas > DE-SOZA/A/EDF/FR) > Objet : Re: [petsc-users] R?f. : Re: Avoiding malloc > overhead for unstructured finite element meshes > > > > On Thu, Jun 28, 2012 at 9:29 AM, Thomas DE-SOZA <*thomas.de-soza at edf.fr*> > wrote: > Since 9900 coefficients were uneeded, we had first thought that enough > room was preallocated. > From what you're telling us, I understand that we may have given an > overall size which is large enough to contain the diagonal block but whose > nnz line by line is not correct hence the mallocs. Is that correct ? > > Unlikely, extra space is dynamically allocated in bigger chunks so there > is usually some left over when mallocs are needed in MatSetValues. You > should be careful to get nnz correct, but that is not the only problem here. > > > Or does it mean that in the preallocation we have to take care of the > values that come from the stash of another processor even if they are added > to preexisting entries on the process ? > > You only have to allocate the entries once, but there are entries that > were not preallocated at all. > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From popov at uni-mainz.de Fri Jun 29 05:05:39 2012 From: popov at uni-mainz.de (Anton Popov) Date: Fri, 29 Jun 2012 12:05:39 +0200 Subject: [petsc-users] fieldsplit recursive In-Reply-To: References: <4FEC7BCD.7030803@uni-mainz.de> Message-ID: <4FED7DF3.6060203@uni-mainz.de> Thanks Barry, We'll send the code to petsc-maint in a moment. But we really don't want to bother you a lot. Anton and Boris On 6/28/12 8:55 PM, Barry Smith wrote: > Anton, > > This came about because we are now being much more pedantic about the blocksizes of PETSc objects and not allowing them to be causally changed when they shouldn't be. > > You can resolve this problem by editing the file src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function > > #undef __FUNCT__ > #define __FUNCT__ "PCApply_FieldSplit" > static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) > { > PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; > PetscErrorCode ierr; > PC_FieldSplitLink ilink = jac->head; > PetscInt cnt,bs; > > PetscFunctionBegin; > > and add the two lines right here > > x->map->bs = jac->bs; > y->map->bs = jac->bs; > > > then run make cmake in that directory. > > To resolve this permanently we will need to figure out how to insure those inner vectors are created with the correct block size. Are you willing to share your code with petsc-maint at mcs.anl.gov so that we can reproduce the problem and fix it properly for the long run? (The problem is in PETSc not in your code). > > Barry > > > > On Jun 28, 2012, at 10:44 AM, Anton Popov wrote: > >> Dear petsc team, >> >> I'm trying to use fieldsplit preconditioner for the velocity block in the Stokes system which is also preconditioned by >> fieldsplit (kind of recursive). >> >> Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as follows: >> >> mpiexec -np 4 ./ex42 \ >> -stokes_ksp_type gcr \ >> -stokes_ksp_rtol 1.0e-6 \ >> -stokes_pc_type fieldsplit \ >> -stokes_pc_fieldsplit_type SCHUR \ >> -stokes_pc_fieldsplit_schur_factorization_type UPPER \ >> -stokes_fieldsplit_u_ksp_type fgmres \ >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ >> -stokes_fieldsplit_u_pc_type fieldsplit \ >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ >> -stokes_fieldsplit_p_ksp_type preonly \ >> -stokes_fieldsplit_p_pc_type jacobi \ >> -stokes_ksp_monitor_blocks \ >> -mx 16 \ >> -model 3 >> >> gives nicely looking output. >> >> But! Repeating the same exercise with petsc-3.3, like this (actually, there is only one difference: factorization -> fact): >> >> mpiexec -np 4 ./ex42 \ >> -stokes_ksp_type gcr \ >> -stokes_ksp_rtol 1.0e-6 \ >> -stokes_pc_type fieldsplit \ >> -stokes_pc_fieldsplit_type SCHUR \ >> -stokes-pc_fieldsplit_schur_fact_type UPPER \ >> -stokes_fieldsplit_u_ksp_type fgmres \ >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ >> -stokes_fieldsplit_u_pc_type fieldsplit \ >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ >> -stokes_fieldsplit_p_ksp_type preonly \ >> -stokes_fieldsplit_p_pc_type jacobi \ >> -stokes_ksp_monitor_blocks \ >> -mx 16 \ >> -model 3 >> >> curses me hardly by claiming: >> >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 3! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by anton Thu Jun 28 17:06:53 2012 >> [0]PETSC ERROR: Libraries linked from /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib >> [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 >> [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCApply_FieldSplit() line 726 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPFGMRESCycle() line 169 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: KSPSolve_GCR() line 117 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in src/ksp/ksp/examples/tutorials/ex42.c >> [0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> Similar error appeared in our code after upgrading to petsc-3.3, and we're using similar functionality and options as I posted above. >> >> Please explain this issue. An advice how to get rid of the error is also appreciated. >> >> Thanks a lot >> >> Anton From kaus at uni-mainz.de Fri Jun 29 03:17:08 2012 From: kaus at uni-mainz.de (Kaus, Boris) Date: Fri, 29 Jun 2012 08:17:08 +0000 Subject: [petsc-users] fieldsplit recursive In-Reply-To: References: <4FEC7BCD.7030803@uni-mainz.de> Message-ID: <65DC2AC1-5082-4A9D-8775-41F99CAE9B70@uni-mainz.de> Hi, If I understand this correctly, it also implies that the fieldsplit-in-fieldsplit options described in Jed's manuscript on the new multiphysics options of PETSc http://www.mcs.anl.gov/uploads/cels/papers/P2017-0112.pdf section IV are broken in PETSC 3.3. That's a bummer as it is the preconditioner we most commonly used. cheers, Boris On Jun 28, 2012, at 8:55 PM, Barry Smith wrote: > Anton, > > This came about because we are now being much more pedantic about the blocksizes of PETSc objects and not allowing them to be causally changed when they shouldn't be. > > You can resolve this problem by editing the file src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function > > #undef __FUNCT__ > #define __FUNCT__ "PCApply_FieldSplit" > static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) > { > PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; > PetscErrorCode ierr; > PC_FieldSplitLink ilink = jac->head; > PetscInt cnt,bs; > > PetscFunctionBegin; > > and add the two lines right here > > x->map->bs = jac->bs; > y->map->bs = jac->bs; > > > then run make cmake in that directory. > > To resolve this permanently we will need to figure out how to insure those inner vectors are created with the correct block size. Are you willing to share your code with petsc-maint at mcs.anl.gov so that we can reproduce the problem and fix it properly for the long run? (The problem is in PETSc not in your code). > > Barry > > > > On Jun 28, 2012, at 10:44 AM, Anton Popov wrote: > >> Dear petsc team, >> >> I'm trying to use fieldsplit preconditioner for the velocity block in the Stokes system which is also preconditioned by >> fieldsplit (kind of recursive). >> >> Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as follows: >> >> mpiexec -np 4 ./ex42 \ >> -stokes_ksp_type gcr \ >> -stokes_ksp_rtol 1.0e-6 \ >> -stokes_pc_type fieldsplit \ >> -stokes_pc_fieldsplit_type SCHUR \ >> -stokes_pc_fieldsplit_schur_factorization_type UPPER \ >> -stokes_fieldsplit_u_ksp_type fgmres \ >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ >> -stokes_fieldsplit_u_pc_type fieldsplit \ >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ >> -stokes_fieldsplit_p_ksp_type preonly \ >> -stokes_fieldsplit_p_pc_type jacobi \ >> -stokes_ksp_monitor_blocks \ >> -mx 16 \ >> -model 3 >> >> gives nicely looking output. >> >> But! Repeating the same exercise with petsc-3.3, like this (actually, there is only one difference: factorization -> fact): >> >> mpiexec -np 4 ./ex42 \ >> -stokes_ksp_type gcr \ >> -stokes_ksp_rtol 1.0e-6 \ >> -stokes_pc_type fieldsplit \ >> -stokes_pc_fieldsplit_type SCHUR \ >> -stokes-pc_fieldsplit_schur_fact_type UPPER \ >> -stokes_fieldsplit_u_ksp_type fgmres \ >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ >> -stokes_fieldsplit_u_pc_type fieldsplit \ >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ >> -stokes_fieldsplit_p_ksp_type preonly \ >> -stokes_fieldsplit_p_pc_type jacobi \ >> -stokes_ksp_monitor_blocks \ >> -mx 16 \ >> -model 3 >> >> curses me hardly by claiming: >> >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: Object is in wrong state! >> [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 3! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by anton Thu Jun 28 17:06:53 2012 >> [0]PETSC ERROR: Libraries linked from /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib >> [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 >> [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCApply_FieldSplit() line 726 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPFGMRESCycle() line 169 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: KSPSolve_GCR() line 117 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in src/ksp/ksp/examples/tutorials/ex42.c >> [0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> Similar error appeared in our code after upgrading to petsc-3.3, and we're using similar functionality and options as I posted above. >> >> Please explain this issue. An advice how to get rid of the error is also appreciated. >> >> Thanks a lot >> >> Anton > ----------------------------------------------------------------------------- Boris J.P. Kaus Institute of Geosciences & Center for Computational Sciences. University of Mainz, Mainz, Germany Office: 00-285 Tel: +49.6131.392.4527 http://www.geophysik.uni-mainz.de ----------------------------------------------------------------------------- From w_ang_temp at 163.com Fri Jun 29 10:46:11 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Fri, 29 Jun 2012 23:46:11 +0800 (CST) Subject: [petsc-users] ill-conditioned problems in PETSc In-Reply-To: References: <1ea007a1.104fd.1382e39d09b.Coremail.w_ang_temp@163.com> <071D9A22-DF69-488C-ADCE-BCEF3563542E@mcs.anl.gov> <32091ff2.24b58.13833525d35.Coremail.w_ang_temp@163.com> Message-ID: <4eb0328b.d786.13838ebc595.Coremail.w_ang_temp@163.com> Thank you very much. And we will have a try on both MUMPS and the traditional preconditioners provided by PETSc. Jim >? 2012-06-28 23:01:44?"Hong Zhang" ??? >Jim, > In fact, I will deal with the large sparse symmetric or unsymmetric (depending on the constitutive >model used for the soil and the formulation of equations) linear system which usually occurs in geotechnical >problems. As you know, In most geotechnical problems, the coefficient matrix can be severely >ill-conditioned, thus calling for the development of robust and efficient preconditioners and iterative methods. > The iterative method and parallel solver are my choices, actually are required by my professor. As I know, >MUMPS is just an external solver like Matlab. So is it a right way to analyse the preconditioners and iterative >MUMPS and Superlu_dist are parallel sparse direct solvers for ill-conditioned problems. >They can solve problems with 10k unknowns sequentially and scale up to >10+ or 100+ processors, i.e. solve problems with 100k unknowns. >Give them a try. Then, explore other preconditioners. >methods provided by PETSc like CG, GMRES, Jacobi, sor, etc. for my work? > Also I do not quite understand the second part of what Barry said ('You will not be able to use the external >solvers with those mode'). So, what does 'those mode' mean. Using quad precision? >Yes, high precisions. Start from small problems with robust solvers, then slowly >move to large, complex problems. >Hong -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 29 11:35:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 29 Jun 2012 08:35:48 -0800 Subject: [petsc-users] fieldsplit recursive In-Reply-To: <65DC2AC1-5082-4A9D-8775-41F99CAE9B70@uni-mainz.de> References: <4FEC7BCD.7030803@uni-mainz.de> <65DC2AC1-5082-4A9D-8775-41F99CAE9B70@uni-mainz.de> Message-ID: On Fri, Jun 29, 2012 at 12:17 AM, Kaus, Boris wrote: > Hi, > > If I understand this correctly, it also implies that the > fieldsplit-in-fieldsplit options described in Jed's manuscript on the new > multiphysics options of PETSc > > http://www.mcs.anl.gov/uploads/cels/papers/P2017-0112.pdf > > section IV are broken in PETSC 3.3. > That's a bummer as it is the preconditioner we most commonly used. > There are nightly tests that use these options (src/ksp/ksp/examples/tests/ex11.c and src/ksp/ksp/examples/tutorials/ex43.c) so it appears that this problem is not always triggered. Thanks for the test case, we'll get this sorted out. > > cheers, > Boris > > > > > > > On Jun 28, 2012, at 8:55 PM, Barry Smith wrote: > > > Anton, > > > > This came about because we are now being much more pedantic about the > blocksizes of PETSc objects and not allowing them to be causally changed > when they shouldn't be. > > > > You can resolve this problem by editing the file > src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function > > > > #undef __FUNCT__ > > #define __FUNCT__ "PCApply_FieldSplit" > > static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) > > { > > PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; > > PetscErrorCode ierr; > > PC_FieldSplitLink ilink = jac->head; > > PetscInt cnt,bs; > > > > PetscFunctionBegin; > > > > and add the two lines right here > > > > x->map->bs = jac->bs; > > y->map->bs = jac->bs; > > > > > > then run make cmake in that directory. > > > > To resolve this permanently we will need to figure out how to insure > those inner vectors are created with the correct block size. Are you > willing to share your code with petsc-maint at mcs.anl.gov so that we can > reproduce the problem and fix it properly for the long run? (The problem is > in PETSc not in your code). > > > > Barry > > > > > > > > On Jun 28, 2012, at 10:44 AM, Anton Popov wrote: > > > >> Dear petsc team, > >> > >> I'm trying to use fieldsplit preconditioner for the velocity block in > the Stokes system which is also preconditioned by > >> fieldsplit (kind of recursive). > >> > >> Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, > as follows: > >> > >> mpiexec -np 4 ./ex42 \ > >> -stokes_ksp_type gcr \ > >> -stokes_ksp_rtol 1.0e-6 \ > >> -stokes_pc_type fieldsplit \ > >> -stokes_pc_fieldsplit_type SCHUR \ > >> -stokes_pc_fieldsplit_schur_factorization_type UPPER \ > >> -stokes_fieldsplit_u_ksp_type fgmres \ > >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > >> -stokes_fieldsplit_u_pc_type fieldsplit \ > >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > >> -stokes_fieldsplit_p_ksp_type preonly \ > >> -stokes_fieldsplit_p_pc_type jacobi \ > >> -stokes_ksp_monitor_blocks \ > >> -mx 16 \ > >> -model 3 > >> > >> gives nicely looking output. > >> > >> But! Repeating the same exercise with petsc-3.3, like this (actually, > there is only one difference: factorization -> fact): > >> > >> mpiexec -np 4 ./ex42 \ > >> -stokes_ksp_type gcr \ > >> -stokes_ksp_rtol 1.0e-6 \ > >> -stokes_pc_type fieldsplit \ > >> -stokes_pc_fieldsplit_type SCHUR \ > >> -stokes-pc_fieldsplit_schur_fact_type UPPER \ > >> -stokes_fieldsplit_u_ksp_type fgmres \ > >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > >> -stokes_fieldsplit_u_pc_type fieldsplit \ > >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > >> -stokes_fieldsplit_p_ksp_type preonly \ > >> -stokes_fieldsplit_p_pc_type jacobi \ > >> -stokes_ksp_monitor_blocks \ > >> -mx 16 \ > >> -model 3 > >> > >> curses me hardly by claiming: > >> > >> [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > >> [0]PETSC ERROR: Object is in wrong state! > >> [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit > blocksize 3! > >> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 > 14:20:42 CDT 2012 > >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [0]PETSC ERROR: See docs/index.html for manual pages. > >> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.deby anton Thu Jun 28 17:06:53 2012 > >> [0]PETSC ERROR: Libraries linked from > /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib > >> [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 > >> [0]PETSC ERROR: Configure options > PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug > --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" > --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 > --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 > --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --download-ml=1 --download-hypre=1 --download-blacs=1 > --download-scalapack=1 --download-metis=1 --download-parmetis=1 > --download-mumps=1 --download-superlu_dist=1 > >> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >> [0]PETSC ERROR: PCApply_FieldSplit() line 726 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: PCApply() line 384 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: KSPFGMRESCycle() line 169 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > >> [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > >> [0]PETSC ERROR: KSPSolve() line 446 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: PCApply() line 384 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > >> [0]PETSC ERROR: KSPSolve_GCR() line 117 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > >> [0]PETSC ERROR: KSPSolve() line 446 in > /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in > src/ksp/ksp/examples/tutorials/ex42.c > >> [0]PETSC ERROR: main() line 2106 in > src/ksp/ksp/examples/tutorials/ex42.c > >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > >> > >> Similar error appeared in our code after upgrading to petsc-3.3, and > we're using similar functionality and options as I posted above. > >> > >> Please explain this issue. An advice how to get rid of the error is > also appreciated. > >> > >> Thanks a lot > >> > >> Anton > > > > > > > ----------------------------------------------------------------------------- > Boris J.P. Kaus > > Institute of Geosciences & > Center for Computational Sciences. > University of Mainz, Mainz, Germany > Office: 00-285 > Tel: +49.6131.392.4527 > > http://www.geophysik.uni-mainz.de > > ----------------------------------------------------------------------------- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bernardomartinsrocha at gmail.com Fri Jun 29 15:20:23 2012 From: bernardomartinsrocha at gmail.com (Bernardo Rocha) Date: Fri, 29 Jun 2012 17:20:23 -0300 Subject: [petsc-users] MatAssemblyBegin-End Message-ID: Hi everyone, I've a simple question: I'm solving successive linear systems, and the way I've implemented my problem it calls MatAssemblyBegin and MatAssemblyEnd every time before the call to KSPSolve. The sparsity pattern of my matrix does not change between the calls. Is it necessary to call the pair MatAssemblyBegin/End every time? If it is not necessary, do these routines check if I've already assembled the matrices or not? Does it hurt performance if I do so? Thanks. Best regards, Bernardo M. R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Jun 29 15:29:40 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 29 Jun 2012 12:29:40 -0800 Subject: [petsc-users] MatAssemblyBegin-End In-Reply-To: References: Message-ID: Call these functions when you assemble a matrix (after MatSetValues). You should call KSPSetOperators() before solving a system with a different matrix. When the matrix is constant, just call KSPSolve() multiple times without anything in between. On Fri, Jun 29, 2012 at 12:20 PM, Bernardo Rocha < bernardomartinsrocha at gmail.com> wrote: > Hi everyone, > > I've a simple question: I'm solving successive linear systems, and the way > I've implemented my problem it calls MatAssemblyBegin and MatAssemblyEnd > every time before the call to KSPSolve. The sparsity pattern of my matrix > does not change between the calls. Is it necessary to call the pair > MatAssemblyBegin/End every time? If it is not necessary, do these routines > check if I've already assembled the matrices or not? Does it hurt > performance if I do so? > > Thanks. > Best regards, > Bernardo M. R. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrosso at uci.edu Fri Jun 29 19:22:10 2012 From: mrosso at uci.edu (Michele Rosso) Date: Fri, 29 Jun 2012 17:22:10 -0700 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver In-Reply-To: References: <4FE9F363.8040506@uci.edu> <4FE9FE6D.5090905@uci.edu> Message-ID: <4FEE46B2.5050903@uci.edu> Hi Jed, thank you for your reply. I checked the PETSc version currently installed on my system (Blue Gene P); it is 3.1-p2 so I will use the DA type. Also, I cannot take advantage of DMDAGlobalToNatural since version 3.1 does not support it. I was able to solve a simple Poisson system successfully but the solution vector is not in the correct order and I do not understand how to fix this. Is there an equivalent to DMDAGlobalToNatural in version 3.1? Thank you, Michele On 06/26/2012 11:36 AM, Jed Brown wrote: > On Tue, Jun 26, 2012 at 10:24 AM, Michele Rosso > wrote: > > Thanks Jed for your reply. > > 1) I will use the DMDA object type then. I am still not very clear > about the difference between DA and DMDA though. > > > DM is the generic interface, DMDA is the structured grid > implementation. It used to be called just DA and managed a bit > different. Maybe you were looking at old docs? > > > 2) I am not interested in having ghost nodes updated. I want the > proper values of the solution on the proper processor. > I have to fill the known-terms-vector with nodes-dependent > values ( in contrast with ex45f.F, where vector b is filled with > 1s, thus there is > no dependence on the grid location). Since every processor > defines "non-local" (according to the PETSc internal ordering) > components, the vector is re-arranged > and so is the solution vector. So I will have on every > process a solution which partially should be on a difference > process. And this is not about ghost cell. > Sorry for this long explanation but I am trying to be as clear > as I can. > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGlobalToNaturalBegin.html > > > Thank you fro your help and patience, > > Michele > > > > > > > > > > On 06/26/2012 10:42 AM, Jed Brown wrote: >> On Tue, Jun 26, 2012 at 9:37 AM, Michele Rosso > > wrote: >> >> Hi, >> >> I need some help to use the PETSc library inside my Fortran >> 95 code. >> My goal is to solve a symmetric linear system that derives >> from the finite difference discretization >> of the Poisson equation. I will use the preconditioned >> conjugate method. >> >> I am mostly interested in how to decompose my data among the >> different processes. >> In particular: >> >> 1) Since my code already implements the 2D-decomposition, >> would it be best to build the matrix with the DMDA object >> type, DA object type >> or the regular Mat type? >> >> >> It is certainly easiest to just use DMDA (and you will get >> geometric multigrid for free, which is unbeatable for this >> problem), but you can work directly with Mat if you prefer. See >> src/ksp/ksp/examples/tutorials/ex45f.F for a Fortran example. >> >> >> 2) After inserting values into a vector/matrix, PETSc >> performs any needed message passing of nonlocal components, >> thus the values locally contained on a process may be >> communicated to another process. How can I revert this at >> the end of the computation, that is, how can I be sure that >> the local solution vector contains the values associated to >> the grid nodes contained into the hosting process? >> >> >> Please read the section of the user's manual on local versus >> global spaces and the structured grid decompositions. If you use >> DM, there is DMGlobalToLocalBegin/End that update the ghost >> points in the local vectors. >> >> >> >> Thank you, >> >> Michele >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jun 29 19:38:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 29 Jun 2012 18:38:47 -0600 Subject: [petsc-users] Data decomposition in PETsc for Poisson solver In-Reply-To: <4FEE46B2.5050903@uci.edu> References: <4FE9F363.8040506@uci.edu> <4FE9FE6D.5090905@uci.edu> <4FEE46B2.5050903@uci.edu> Message-ID: On Fri, Jun 29, 2012 at 6:22 PM, Michele Rosso wrote: > Hi Jed, > > thank you for your reply. > I checked the PETSc version currently installed on my system (Blue Gene > P); it is 3.1-p2 > so I will use the DA type. > Also, I cannot take advantage of DMDAGlobalToNatural since version 3.1 > does not support it. > I was able to solve a simple Poisson system successfully but the solution > vector is not in the correct order > and I do not understand how to fix this. Is there an equivalent to DMDAGlobalToNatural > in version 3.1? > The code is here: http://www.mcs.anl.gov/petsc/petsc-current/src/dm/impls/da/dagtol.c.html#DMDANaturalToGlobalBegin It should work in 3.1 with minor alterations. Otherwise, you can build your own 3.3. Matt > Thank you, > Michele > > > On 06/26/2012 11:36 AM, Jed Brown wrote: > > On Tue, Jun 26, 2012 at 10:24 AM, Michele Rosso wrote: > >> Thanks Jed for your reply. >> >> 1) I will use the DMDA object type then. I am still not very clear about >> the difference between DA and DMDA though. >> > > DM is the generic interface, DMDA is the structured grid implementation. > It used to be called just DA and managed a bit different. Maybe you were > looking at old docs? > > >> >> 2) I am not interested in having ghost nodes updated. I want the proper >> values of the solution on the proper processor. >> I have to fill the known-terms-vector with nodes-dependent values ( >> in contrast with ex45f.F, where vector b is filled with 1s, thus there >> is >> no dependence on the grid location). Since every processor defines >> "non-local" (according to the PETSc internal ordering) components, the >> vector is re-arranged >> and so is the solution vector. So I will have on every process a >> solution which partially should be on a difference process. And this is >> not about ghost cell. >> Sorry for this long explanation but I am trying to be as clear as I >> can. >> > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAGlobalToNaturalBegin.html > > >> >> Thank you fro your help and patience, >> >> Michele >> >> >> >> >> >> >> >> >> >> On 06/26/2012 10:42 AM, Jed Brown wrote: >> >> On Tue, Jun 26, 2012 at 9:37 AM, Michele Rosso wrote: >> >>> Hi, >>> >>> I need some help to use the PETSc library inside my Fortran 95 code. >>> My goal is to solve a symmetric linear system that derives from the >>> finite difference discretization >>> of the Poisson equation. I will use the preconditioned conjugate method. >>> >>> I am mostly interested in how to decompose my data among the different >>> processes. >>> In particular: >>> >>> 1) Since my code already implements the 2D-decomposition, would it be >>> best to build the matrix with the DMDA object type, DA object type >>> or the regular Mat type? >>> >> >> It is certainly easiest to just use DMDA (and you will get geometric >> multigrid for free, which is unbeatable for this problem), but you can work >> directly with Mat if you prefer. See src/ksp/ksp/examples/tutorials/ex45f.F >> for a Fortran example. >> >> >>> >>> 2) After inserting values into a vector/matrix, PETSc performs any >>> needed message passing of nonlocal components, thus the values locally >>> contained on a process may be communicated to another process. How can I >>> revert this at the end of the computation, that is, how can I be sure that >>> the local solution vector contains the values associated to the grid nodes >>> contained into the hosting process? >>> >> >> Please read the section of the user's manual on local versus global >> spaces and the structured grid decompositions. If you use DM, there is >> DMGlobalToLocalBegin/End that update the ghost points in the local vectors. >> >> >>> >>> >>> Thank you, >>> >>> Michele >>> >> >> >> >> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjm2176 at columbia.edu Fri Jun 29 19:46:57 2012 From: cjm2176 at columbia.edu (Colin McAuliffe) Date: Fri, 29 Jun 2012 20:46:57 -0400 Subject: [petsc-users] Replacing values in a Vec Message-ID: <20120629204657.3d3ekn19icc80gwc@cubmail.cc.columbia.edu> Hello, I would like to replace all the entries of a Vec which have a certain value with a different value. Is there a command for this? Thanks Colin -- Colin McAuliffe PhD Candidate Columbia University Department of Civil Engineering and Engineering Mechanics From bsmith at mcs.anl.gov Fri Jun 29 20:17:20 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Jun 2012 20:17:20 -0500 Subject: [petsc-users] Replacing values in a Vec In-Reply-To: <20120629204657.3d3ekn19icc80gwc@cubmail.cc.columbia.edu> References: <20120629204657.3d3ekn19icc80gwc@cubmail.cc.columbia.edu> Message-ID: Just use VecGetArray() then loop over that, checking each entry to see if it matches a "certain value" if so put in the other value. Barry On Jun 29, 2012, at 7:46 PM, Colin McAuliffe wrote: > Hello, > > I would like to replace all the entries of a Vec which have a certain value with a different value. Is there a command for this? > > Thanks > Colin > > > > > -- > Colin McAuliffe > PhD Candidate > Columbia University > Department of Civil Engineering and Engineering Mechanics From renzhengyong at gmail.com Fri Jun 29 20:25:46 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Sat, 30 Jun 2012 03:25:46 +0200 Subject: [petsc-users] about parallel preconditioned matrix-free gmres Message-ID: Dear Petscs, Use the uniprocessor complex-value based version petsc, I recently successfully make a FETI_DP domain decomposition approach working for 3D electromagnetic induction (earth) problem. The number of iteration of the interface problem seems to be scalable with regard to the number of sub-domains. To do this, I had two subroutines for petsc (1) int mat_vec_product_interface_problem(Mat A, Vec X, Vec Y) for matrix-free GMRES solver (2) int preconditioner_mat_vec(PC pc,Vec X,Vec Y) for shell preconditioner. Now, I want to solve the interface problem by paralleled GMRES solver so that I can solve real large-scale problems. Could you please tell me the easiest way to accomplish it. Which specific data structures of petsc should be used. I have been using Petsc for 3.5 years, I really want to have a try the real MPI-based Petsc. Thanks in advance. Have a nice weekeed Zhengyong -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Jun 29 22:07:21 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 29 Jun 2012 22:07:21 -0500 Subject: [petsc-users] fieldsplit recursive In-Reply-To: References: <4FEC7BCD.7030803@uni-mainz.de> <65DC2AC1-5082-4A9D-8775-41F99CAE9B70@uni-mainz.de> Message-ID: <5499CF95-909D-4076-8DCF-56CF7CB14D51@mcs.anl.gov> On Jun 29, 2012, at 11:35 AM, Jed Brown wrote: > On Fri, Jun 29, 2012 at 12:17 AM, Kaus, Boris wrote: > Hi, > > If I understand this correctly, it also implies that the fieldsplit-in-fieldsplit options described in Jed's manuscript on the new multiphysics options of PETSc > > http://www.mcs.anl.gov/uploads/cels/papers/P2017-0112.pdf > > section IV are broken in PETSC 3.3. > That's a bummer as it is the preconditioner we most commonly used. > > There are nightly tests that use these options (src/ksp/ksp/examples/tests/ex11.c and src/ksp/ksp/examples/tutorials/ex43.c) so it appears that this problem is not always triggered. Thanks for the test case, we'll get this sorted out. As Jed says, not really broken but care must be taken with block sizes through the recursion; your code will help us figure it out. Barry > > > cheers, > Boris > > > > > > > On Jun 28, 2012, at 8:55 PM, Barry Smith wrote: > > > Anton, > > > > This came about because we are now being much more pedantic about the blocksizes of PETSc objects and not allowing them to be causally changed when they shouldn't be. > > > > You can resolve this problem by editing the file src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function > > > > #undef __FUNCT__ > > #define __FUNCT__ "PCApply_FieldSplit" > > static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) > > { > > PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; > > PetscErrorCode ierr; > > PC_FieldSplitLink ilink = jac->head; > > PetscInt cnt,bs; > > > > PetscFunctionBegin; > > > > and add the two lines right here > > > > x->map->bs = jac->bs; > > y->map->bs = jac->bs; > > > > > > then run make cmake in that directory. > > > > To resolve this permanently we will need to figure out how to insure those inner vectors are created with the correct block size. Are you willing to share your code with petsc-maint at mcs.anl.gov so that we can reproduce the problem and fix it properly for the long run? (The problem is in PETSc not in your code). > > > > Barry > > > > > > > > On Jun 28, 2012, at 10:44 AM, Anton Popov wrote: > > > >> Dear petsc team, > >> > >> I'm trying to use fieldsplit preconditioner for the velocity block in the Stokes system which is also preconditioned by > >> fieldsplit (kind of recursive). > >> > >> Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as follows: > >> > >> mpiexec -np 4 ./ex42 \ > >> -stokes_ksp_type gcr \ > >> -stokes_ksp_rtol 1.0e-6 \ > >> -stokes_pc_type fieldsplit \ > >> -stokes_pc_fieldsplit_type SCHUR \ > >> -stokes_pc_fieldsplit_schur_factorization_type UPPER \ > >> -stokes_fieldsplit_u_ksp_type fgmres \ > >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > >> -stokes_fieldsplit_u_pc_type fieldsplit \ > >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > >> -stokes_fieldsplit_p_ksp_type preonly \ > >> -stokes_fieldsplit_p_pc_type jacobi \ > >> -stokes_ksp_monitor_blocks \ > >> -mx 16 \ > >> -model 3 > >> > >> gives nicely looking output. > >> > >> But! Repeating the same exercise with petsc-3.3, like this (actually, there is only one difference: factorization -> fact): > >> > >> mpiexec -np 4 ./ex42 \ > >> -stokes_ksp_type gcr \ > >> -stokes_ksp_rtol 1.0e-6 \ > >> -stokes_pc_type fieldsplit \ > >> -stokes_pc_fieldsplit_type SCHUR \ > >> -stokes-pc_fieldsplit_schur_fact_type UPPER \ > >> -stokes_fieldsplit_u_ksp_type fgmres \ > >> -stokes_fieldsplit_u_ksp_rtol 1e-3 \ > >> -stokes_fieldsplit_u_pc_type fieldsplit \ > >> -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \ > >> -stokes_fieldsplit_u_fieldsplit_pc_type ml \ > >> -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \ > >> -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \ > >> -stokes_fieldsplit_p_ksp_type preonly \ > >> -stokes_fieldsplit_p_pc_type jacobi \ > >> -stokes_ksp_monitor_blocks \ > >> -mx 16 \ > >> -model 3 > >> > >> curses me hardly by claiming: > >> > >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >> [0]PETSC ERROR: Object is in wrong state! > >> [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 3! > >> [0]PETSC ERROR: ------------------------------------------------------------------------ > >> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun 5 14:20:42 CDT 2012 > >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. > >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > >> [0]PETSC ERROR: See docs/index.html for manual pages. > >> [0]PETSC ERROR: ------------------------------------------------------------------------ > >> [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by anton Thu Jun 28 17:06:53 2012 > >> [0]PETSC ERROR: Libraries linked from /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib > >> [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012 > >> [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 --download-scalapack=1 --download-metis=1 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1 > >> [0]PETSC ERROR: ------------------------------------------------------------------------ > >> [0]PETSC ERROR: PCApply_FieldSplit() line 726 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: KSPFGMRESCycle() line 169 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > >> [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c > >> [0]PETSC ERROR: PCApply() line 384 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c > >> [0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > >> [0]PETSC ERROR: KSPSolve_GCR() line 117 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c > >> [0]PETSC ERROR: KSPSolve() line 446 in /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c > >> [0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in src/ksp/ksp/examples/tutorials/ex42.c > >> [0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c > >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > >> > >> Similar error appeared in our code after upgrading to petsc-3.3, and we're using similar functionality and options as I posted above. > >> > >> Please explain this issue. An advice how to get rid of the error is also appreciated. > >> > >> Thanks a lot > >> > >> Anton > > > > > > ----------------------------------------------------------------------------- > Boris J.P. Kaus > > Institute of Geosciences & > Center for Computational Sciences. > University of Mainz, Mainz, Germany > Office: 00-285 > Tel: +49.6131.392.4527 > > http://www.geophysik.uni-mainz.de > ----------------------------------------------------------------------------- > > From paeanball at gmail.com Sat Jun 30 03:16:39 2012 From: paeanball at gmail.com (Bao Kai) Date: Sat, 30 Jun 2012 11:16:39 +0300 Subject: [petsc-users] Any suggestions for improving the convergence rate or performance? Message-ID: Hi, all, I am currently trying to solve a solute transport problem with PETSC and finite difference method with structured grids. Two equations are involved, one is a transport equation with advection and diffusion terms, another one is the flow equation with gravity, which is actually a poisson equation. Both the equations use the same solver. Since we eventually want to handle non-linear problems, we us SNES here to solve the equations. I use the BICGSTAB ksp linear solver and block Jacobi preconditioner. All the tolerances are the default ones from PETSC. The range of the solution values for transport equation is about 0-1100, and the range of the solution values of the flow equations should be between -5000-5000. The problem is that the solution of flow equations takes about ten times of the time than the transport equations. I am wondering if I should set different tolerances for the two equations. Attached please find the convergence processes of the two equations. Generally it takes about ten times of iterations to solve the flow equations than transport equations. I only do applications and do not have good knowledge in the solution of linear system, so I am here to find some suggestions on this issue. The size of the mesh is 500X500X500 (125 million unknowns ) and 512 nodes (2048 processors ) on BlueGene/P machine are used. Any suggestions will be much appreciated. Thank you . Kai -------------- next part -------------- RST runs on 2048 processors. Start the simulation of RST_advection on a 3D rectanguluar mesh of size 500x500x500. For Transport Equations: the KSP type is bcgs the PC type is bjacobi KSP rtol = 0.100000000000000008E-04 abstol = 0.100000000000000001E-49 dtol = 10000.0000000000000 maxit = 10000 SNES rtol = 0.100000000000000002E-07 abstol = 0.100000000000000001E-49 stol = 0.100000000000000002E-07 maxit = 50 maxf= 10000 0 SNES Function norm 3.158327432611e+07 0 KSP Residual norm 3.111789893139e+07 0 KSP preconditioned resid norm 3.111789893139e+07 true resid norm 3.158327432611e+07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 3.108810149014e+04 1 KSP preconditioned resid norm 3.108810149014e+04 true resid norm 3.597468808656e+04 ||r(i)||/||b|| 1.139042384115e-03 2 KSP Residual norm 4.118504714081e+01 2 KSP preconditioned resid norm 4.118504714081e+01 true resid norm 5.471372614232e+01 ||r(i)||/||b|| 1.732363958764e-06 1 SNES Function norm 5.471372583447e+01 0 KSP Residual norm 4.118504673314e+01 0 KSP preconditioned resid norm 4.118504673314e+01 true resid norm 5.471372583447e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 5.883751065988e-01 1 KSP preconditioned resid norm 5.883751065988e-01 true resid norm 8.193657571252e-01 ||r(i)||/||b|| 1.497550650460e-02 2 KSP Residual norm 5.154542309080e-03 2 KSP preconditioned resid norm 5.154542309080e-03 true resid norm 7.092044481297e-03 ||r(i)||/||b|| 1.296209383136e-04 3 KSP Residual norm 3.275443113521e-05 3 KSP preconditioned resid norm 3.275443113521e-05 true resid norm 4.412932277080e-05 ||r(i)||/||b|| 8.065494004979e-07 2 SNES Function norm 4.412932277584e-05 Number of KSP iteration is 3 SNES solve takes time 6.68558010117646973 Number of Newton iterations = 2 the SNES Converged Reason is 3 the memory used finally in SolveNE_option is 1046126592.00000000 in 0 th processor solve_NE for concentration equations took time 6.69044996235288636 For flow equations: the KSP type is bcgs the PC type is bjacobi KSP rtol = 0.100000000000000008E-04 abstol = 0.100000000000000001E-49 dtol = 10000.0000000000000 maxit = 10000 SNES rtol = 0.100000000000000002E-07 abstol = 0.100000000000000001E-49 stol = 0.100000000000000002E-07 maxit = 50 maxf= 10000 0 SNES Function norm 1.251113687948e+03 0 KSP Residual norm 2.275248493611e+05 0 KSP preconditioned resid norm 2.275248493611e+05 true resid norm 1.251113687948e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 3.252774532585e+04 1 KSP preconditioned resid norm 3.252774532585e+04 true resid norm 1.201827309292e+02 ||r(i)||/||b|| 9.606059951781e-02 2 KSP Residual norm 1.602764844430e+04 2 KSP preconditioned resid norm 1.602764844430e+04 true resid norm 4.750865103921e+01 ||r(i)||/||b|| 3.797308869438e-02 3 KSP Residual norm 1.132489396868e+04 3 KSP preconditioned resid norm 1.132489396868e+04 true resid norm 3.040333854692e+01 ||r(i)||/||b|| 2.430101983520e-02 4 KSP Residual norm 9.259046239631e+03 4 KSP preconditioned resid norm 9.259046239631e+03 true resid norm 2.708828370917e+01 ||r(i)||/||b|| 2.165133670114e-02 5 KSP Residual norm 7.933147253989e+03 5 KSP preconditioned resid norm 7.933147253989e+03 true resid norm 2.082524388565e+01 ||r(i)||/||b|| 1.664536491468e-02 6 KSP Residual norm 7.225929674833e+03 6 KSP preconditioned resid norm 7.225929674833e+03 true resid norm 1.931333417556e+01 ||r(i)||/||b|| 1.543691381655e-02 7 KSP Residual norm 6.776440536224e+03 7 KSP preconditioned resid norm 6.776440536224e+03 true resid norm 1.918221518615e+01 ||r(i)||/||b|| 1.533211199824e-02 8 KSP Residual norm 6.386730889063e+03 8 KSP preconditioned resid norm 6.386730889063e+03 true resid norm 1.793583729834e+01 ||r(i)||/||b|| 1.433589726586e-02 9 KSP Residual norm 6.119312918996e+03 9 KSP preconditioned resid norm 6.119312918996e+03 true resid norm 1.627528694624e+01 ||r(i)||/||b|| 1.300863950496e-02 10 KSP Residual norm 5.890053054968e+03 10 KSP preconditioned resid norm 5.890053054968e+03 true resid norm 1.690288241329e+01 ||r(i)||/||b|| 1.351026895166e-02 11 KSP Residual norm 5.701864976894e+03 11 KSP preconditioned resid norm 5.701864976894e+03 true resid norm 1.474197957189e+01 ||r(i)||/||b|| 1.178308551325e-02 12 KSP Residual norm 5.552737810616e+03 12 KSP preconditioned resid norm 5.552737810616e+03 true resid norm 1.543846597628e+01 ||r(i)||/||b|| 1.233977865081e-02 13 KSP Residual norm 5.427526642774e+03 13 KSP preconditioned resid norm 5.427526642774e+03 true resid norm 1.377413650084e+01 ||r(i)||/||b|| 1.100950028245e-02 14 KSP Residual norm 5.339080936238e+03 14 KSP preconditioned resid norm 5.339080936238e+03 true resid norm 1.372937508584e+01 ||r(i)||/||b|| 1.097372302620e-02 15 KSP Residual norm 5.250206442958e+03 15 KSP preconditioned resid norm 5.250206442958e+03 true resid norm 1.355200378860e+01 ||r(i)||/||b|| 1.083195229910e-02 16 KSP Residual norm 5.158253203963e+03 16 KSP preconditioned resid norm 5.158253203963e+03 true resid norm 1.338448747124e+01 ||r(i)||/||b|| 1.069805853791e-02 17 KSP Residual norm 5.077303345153e+03 17 KSP preconditioned resid norm 5.077303345153e+03 true resid norm 1.275478312777e+01 ||r(i)||/||b|| 1.019474349185e-02 18 KSP Residual norm 4.998588019932e+03 18 KSP preconditioned resid norm 4.998588019932e+03 true resid norm 1.301308603928e+01 ||r(i)||/||b|| 1.040120187688e-02 19 KSP Residual norm 4.927440973942e+03 19 KSP preconditioned resid norm 4.927440973942e+03 true resid norm 1.213120696813e+01 ||r(i)||/||b|| 9.696326628816e-03 20 KSP Residual norm 4.865777058660e+03 20 KSP preconditioned resid norm 4.865777058660e+03 true resid norm 1.228424058863e+01 ||r(i)||/||b|| 9.818644546028e-03 21 KSP Residual norm 4.797364474479e+03 21 KSP preconditioned resid norm 4.797364474479e+03 true resid norm 1.186596978858e+01 ||r(i)||/||b|| 9.484325767419e-03 22 KSP Residual norm 4.708717613701e+03 22 KSP preconditioned resid norm 4.708717613701e+03 true resid norm 1.228796719231e+01 ||r(i)||/||b|| 9.821623175156e-03 23 KSP Residual norm 4.605272133902e+03 23 KSP preconditioned resid norm 4.605272133902e+03 true resid norm 1.158887996931e+01 ||r(i)||/||b|| 9.262851234818e-03 24 KSP Residual norm 4.497970381365e+03 24 KSP preconditioned resid norm 4.497970381365e+03 true resid norm 1.168631241829e+01 ||r(i)||/||b|| 9.340727809839e-03 25 KSP Residual norm 4.387901968178e+03 25 KSP preconditioned resid norm 4.387901968178e+03 true resid norm 1.132982893608e+01 ||r(i)||/||b|| 9.055794885162e-03 26 KSP Residual norm 4.281274517233e+03 26 KSP preconditioned resid norm 4.281274517233e+03 true resid norm 1.166120212776e+01 ||r(i)||/||b|| 9.320657459107e-03 27 KSP Residual norm 4.185845080260e+03 27 KSP preconditioned resid norm 4.185845080260e+03 true resid norm 1.145192141502e+01 ||r(i)||/||b|| 9.153381923113e-03 28 KSP Residual norm 4.113707532607e+03 28 KSP preconditioned resid norm 4.113707532607e+03 true resid norm 1.170132924134e+01 ||r(i)||/||b|| 9.352730574409e-03 29 KSP Residual norm 4.064712367841e+03 29 KSP preconditioned resid norm 4.064712367841e+03 true resid norm 1.165343088333e+01 ||r(i)||/||b|| 9.314445997663e-03 30 KSP Residual norm 4.031740082814e+03 30 KSP preconditioned resid norm 4.031740082814e+03 true resid norm 1.177142620249e+01 ||r(i)||/||b|| 9.408758225480e-03 31 KSP Residual norm 4.007985759641e+03 31 KSP preconditioned resid norm 4.007985759641e+03 true resid norm 1.157163981465e+01 ||r(i)||/||b|| 9.249071388249e-03 32 KSP Residual norm 3.987946148351e+03 32 KSP preconditioned resid norm 3.987946148351e+03 true resid norm 1.150455928097e+01 ||r(i)||/||b|| 9.195454731082e-03 33 KSP Residual norm 3.967598294565e+03 33 KSP preconditioned resid norm 3.967598294565e+03 true resid norm 1.117496362312e+01 ||r(i)||/||b|| 8.932012918387e-03 34 KSP Residual norm 3.949602307171e+03 34 KSP preconditioned resid norm 3.949602307171e+03 true resid norm 1.104570613289e+01 ||r(i)||/||b|| 8.828698973795e-03 35 KSP Residual norm 3.947973257592e+03 35 KSP preconditioned resid norm 3.947973257592e+03 true resid norm 1.055187218883e+01 ||r(i)||/||b|| 8.433983490450e-03 36 KSP Residual norm 3.962156854583e+03 36 KSP preconditioned resid norm 3.962156854583e+03 true resid norm 1.053124682653e+01 ||r(i)||/||b|| 8.417497888458e-03 37 KSP Residual norm 3.990239707303e+03 37 KSP preconditioned resid norm 3.990239707303e+03 true resid norm 1.013666605864e+01 ||r(i)||/||b|| 8.102114265299e-03 38 KSP Residual norm 4.021459130706e+03 38 KSP preconditioned resid norm 4.021459130706e+03 true resid norm 1.022737422520e+01 ||r(i)||/||b|| 8.174616202923e-03 39 KSP Residual norm 4.033386536602e+03 39 KSP preconditioned resid norm 4.033386536602e+03 true resid norm 9.889107716319e+00 ||r(i)||/||b|| 7.904243884132e-03 40 KSP Residual norm 4.001694656251e+03 40 KSP preconditioned resid norm 4.001694656251e+03 true resid norm 1.127454443678e+01 ||r(i)||/||b|| 9.011606655244e-03 41 KSP Residual norm 4.243914785438e+03 41 KSP preconditioned resid norm 4.243914785438e+03 true resid norm 1.058001675647e+01 ||r(i)||/||b|| 8.456479102085e-03 42 KSP Residual norm 4.162857908768e+03 42 KSP preconditioned resid norm 4.162857908768e+03 true resid norm 1.124258804253e+01 ||r(i)||/||b|| 8.986064296814e-03 43 KSP Residual norm 3.765389870786e+03 43 KSP preconditioned resid norm 3.765389870786e+03 true resid norm 1.249032062367e+01 ||r(i)||/||b|| 9.983361819146e-03 44 KSP Residual norm 3.650952006957e+03 44 KSP preconditioned resid norm 3.650952006957e+03 true resid norm 8.729106682918e+00 ||r(i)||/||b|| 6.977069124097e-03 45 KSP Residual norm 3.296530027654e+03 45 KSP preconditioned resid norm 3.296530027654e+03 true resid norm 1.321064076628e+01 ||r(i)||/||b|| 1.055910497466e-02 46 KSP Residual norm 3.078881988928e+03 46 KSP preconditioned resid norm 3.078881988928e+03 true resid norm 7.540995792770e+00 ||r(i)||/||b|| 6.027426496418e-03 47 KSP Residual norm 3.072913066195e+03 47 KSP preconditioned resid norm 3.072913066195e+03 true resid norm 7.525988357042e+00 ||r(i)||/||b|| 6.015431235018e-03 48 KSP Residual norm 3.054495479726e+03 48 KSP preconditioned resid norm 3.054495479726e+03 true resid norm 8.060599149405e+00 ||r(i)||/||b|| 6.442739158761e-03 49 KSP Residual norm 3.032937903582e+03 49 KSP preconditioned resid norm 3.032937903582e+03 true resid norm 7.451105306190e+00 ||r(i)||/||b|| 5.955578120490e-03 50 KSP Residual norm 3.025268720054e+03 50 KSP preconditioned resid norm 3.025268720054e+03 true resid norm 7.545175899452e+00 ||r(i)||/||b|| 6.030767605002e-03 51 KSP Residual norm 2.955008251022e+03 51 KSP preconditioned resid norm 2.955008251022e+03 true resid norm 1.013176571519e+01 ||r(i)||/||b|| 8.098197480200e-03 52 KSP Residual norm 2.859956114214e+03 52 KSP preconditioned resid norm 2.859956114214e+03 true resid norm 7.109419310328e+00 ||r(i)||/||b|| 5.682472647220e-03 53 KSP Residual norm 2.723198176338e+03 53 KSP preconditioned resid norm 2.723198176338e+03 true resid norm 9.721063702529e+00 ||r(i)||/||b|| 7.769928341582e-03 54 KSP Residual norm 2.631127850450e+03 54 KSP preconditioned resid norm 2.631127850450e+03 true resid norm 6.573134188629e+00 ||r(i)||/||b|| 5.253826452342e-03 55 KSP Residual norm 2.527024708521e+03 55 KSP preconditioned resid norm 2.527024708521e+03 true resid norm 6.643100046257e+00 ||r(i)||/||b|| 5.309749313950e-03 56 KSP Residual norm 2.482781210041e+03 56 KSP preconditioned resid norm 2.482781210041e+03 true resid norm 6.776817686509e+00 ||r(i)||/||b|| 5.416628202368e-03 57 KSP Residual norm 2.465342801667e+03 57 KSP preconditioned resid norm 2.465342801667e+03 true resid norm 6.209073327876e+00 ||r(i)||/||b|| 4.962837020877e-03 58 KSP Residual norm 2.460044172941e+03 58 KSP preconditioned resid norm 2.460044172941e+03 true resid norm 6.519781728173e+00 ||r(i)||/||b|| 5.211182477641e-03 59 KSP Residual norm 2.438180920132e+03 59 KSP preconditioned resid norm 2.438180920132e+03 true resid norm 6.145767213086e+00 ||r(i)||/||b|| 4.912237210964e-03 60 KSP Residual norm 2.436269197320e+03 60 KSP preconditioned resid norm 2.436269197320e+03 true resid norm 6.111453279017e+00 ||r(i)||/||b|| 4.884810499548e-03 61 KSP Residual norm 2.406305905915e+03 61 KSP preconditioned resid norm 2.406305905915e+03 true resid norm 6.435343874485e+00 ||r(i)||/||b|| 5.143692325067e-03 62 KSP Residual norm 2.425667449339e+03 62 KSP preconditioned resid norm 2.425667449339e+03 true resid norm 6.217218708667e+00 ||r(i)||/||b|| 4.969347524975e-03 63 KSP Residual norm 2.390727336326e+03 63 KSP preconditioned resid norm 2.390727336326e+03 true resid norm 6.140142079942e+00 ||r(i)||/||b|| 4.907741110252e-03 64 KSP Residual norm 2.379550106193e+03 64 KSP preconditioned resid norm 2.379550106193e+03 true resid norm 5.974333315408e+00 ||r(i)||/||b|| 4.775212175327e-03 65 KSP Residual norm 2.375316122300e+03 65 KSP preconditioned resid norm 2.375316122300e+03 true resid norm 5.952287559558e+00 ||r(i)||/||b|| 4.757591269999e-03 66 KSP Residual norm 2.369998880104e+03 66 KSP preconditioned resid norm 2.369998880104e+03 true resid norm 6.117005731527e+00 ||r(i)||/||b|| 4.889248507511e-03 67 KSP Residual norm 2.361800098212e+03 67 KSP preconditioned resid norm 2.361800098212e+03 true resid norm 5.933810560163e+00 ||r(i)||/||b|| 4.742822828431e-03 68 KSP Residual norm 2.356358292125e+03 68 KSP preconditioned resid norm 2.356358292125e+03 true resid norm 5.921277900081e+00 ||r(i)||/||b|| 4.732805625196e-03 69 KSP Residual norm 2.399298824622e+03 69 KSP preconditioned resid norm 2.399298824622e+03 true resid norm 6.019849295683e+00 ||r(i)||/||b|| 4.811592546442e-03 70 KSP Residual norm 2.347612941650e+03 70 KSP preconditioned resid norm 2.347612941650e+03 true resid norm 5.922925544777e+00 ||r(i)||/||b|| 4.734122567623e-03 71 KSP Residual norm 2.344339088397e+03 71 KSP preconditioned resid norm 2.344339088397e+03 true resid norm 5.893850722646e+00 ||r(i)||/||b|| 4.710883414849e-03 72 KSP Residual norm 2.335775255289e+03 72 KSP preconditioned resid norm 2.335775255289e+03 true resid norm 5.845869253840e+00 ||r(i)||/||b|| 4.672532408647e-03 73 KSP Residual norm 2.311269445792e+03 73 KSP preconditioned resid norm 2.311269445792e+03 true resid norm 6.448864301908e+00 ||r(i)||/||b|| 5.154499038759e-03 74 KSP Residual norm 2.293362844552e+03 74 KSP preconditioned resid norm 2.293362844552e+03 true resid norm 5.733453076129e+00 ||r(i)||/||b|| 4.582679520941e-03 75 KSP Residual norm 2.283790502427e+03 75 KSP preconditioned resid norm 2.283790502427e+03 true resid norm 5.716922129871e+00 ||r(i)||/||b|| 4.569466536048e-03 76 KSP Residual norm 2.272881928008e+03 76 KSP preconditioned resid norm 2.272881928008e+03 true resid norm 6.002513028115e+00 ||r(i)||/||b|| 4.797735877991e-03 77 KSP Residual norm 3.361943629099e+03 77 KSP preconditioned resid norm 3.361943629099e+03 true resid norm 9.575729433737e+00 ||r(i)||/||b|| 7.653764422832e-03 78 KSP Residual norm 2.269539615639e+03 78 KSP preconditioned resid norm 2.269539615639e+03 true resid norm 5.671932891188e+00 ||r(i)||/||b|| 4.533507183100e-03 79 KSP Residual norm 1.822070720926e+03 79 KSP preconditioned resid norm 1.822070720926e+03 true resid norm 5.606269852431e+00 ||r(i)||/||b|| 4.481023512439e-03 80 KSP Residual norm 1.800874022866e+03 80 KSP preconditioned resid norm 1.800874022866e+03 true resid norm 4.681607040475e+00 ||r(i)||/||b|| 3.741951739136e-03 81 KSP Residual norm 1.749658730633e+03 81 KSP preconditioned resid norm 1.749658730633e+03 true resid norm 4.480787796628e+00 ||r(i)||/||b|| 3.581439352627e-03 82 KSP Residual norm 1.708726863588e+03 82 KSP preconditioned resid norm 1.708726863588e+03 true resid norm 4.355570516838e+00 ||r(i)||/||b|| 3.481354699253e-03 83 KSP Residual norm 1.629338817791e+03 83 KSP preconditioned resid norm 1.629338817791e+03 true resid norm 4.070706880505e+00 ||r(i)||/||b|| 3.253666648936e-03 84 KSP Residual norm 1.624694911343e+03 84 KSP preconditioned resid norm 1.624694911343e+03 true resid norm 4.203845791425e+00 ||r(i)||/||b|| 3.360082966016e-03 85 KSP Residual norm 1.580256162337e+03 85 KSP preconditioned resid norm 1.580256162337e+03 true resid norm 3.940463265654e+00 ||r(i)||/||b|| 3.149564506897e-03 86 KSP Residual norm 1.548301145121e+03 86 KSP preconditioned resid norm 1.548301145121e+03 true resid norm 4.066377269611e+00 ||r(i)||/||b|| 3.250206043449e-03 87 KSP Residual norm 1.511982631023e+03 87 KSP preconditioned resid norm 1.511982631023e+03 true resid norm 3.773933143475e+00 ||r(i)||/||b|| 3.016458999553e-03 88 KSP Residual norm 1.467211884720e+03 88 KSP preconditioned resid norm 1.467211884720e+03 true resid norm 4.007100597593e+00 ||r(i)||/||b|| 3.202826918283e-03 89 KSP Residual norm 1.431947899418e+03 89 KSP preconditioned resid norm 1.431947899418e+03 true resid norm 3.629878734313e+00 ||r(i)||/||b|| 2.901318057087e-03 90 KSP Residual norm 1.395324500257e+03 90 KSP preconditioned resid norm 1.395324500257e+03 true resid norm 3.603138801610e+00 ||r(i)||/||b|| 2.879945153121e-03 91 KSP Residual norm 1.373725047495e+03 91 KSP preconditioned resid norm 1.373725047495e+03 true resid norm 3.445803111834e+00 ||r(i)||/||b|| 2.754188644107e-03 92 KSP Residual norm 1.335081950056e+03 92 KSP preconditioned resid norm 1.335081950056e+03 true resid norm 3.343315387483e+00 ||r(i)||/||b|| 2.672271448781e-03 93 KSP Residual norm 1.310623742109e+03 93 KSP preconditioned resid norm 1.310623742109e+03 true resid norm 3.282204409181e+00 ||r(i)||/||b|| 2.623426184845e-03 94 KSP Residual norm 1.310079492713e+03 94 KSP preconditioned resid norm 1.310079492713e+03 true resid norm 3.286822170292e+00 ||r(i)||/||b|| 2.627117105307e-03 95 KSP Residual norm 1.244449671057e+03 95 KSP preconditioned resid norm 1.244449671057e+03 true resid norm 3.132750256608e+00 ||r(i)||/||b|| 2.503969292947e-03 96 KSP Residual norm 1.212865491563e+03 96 KSP preconditioned resid norm 1.212865491563e+03 true resid norm 3.630779553072e+00 ||r(i)||/||b|| 2.902038070598e-03 97 KSP Residual norm 1.189955290819e+03 97 KSP preconditioned resid norm 1.189955290819e+03 true resid norm 2.978110279642e+00 ||r(i)||/||b|| 2.380367434494e-03 98 KSP Residual norm 1.180461366945e+03 98 KSP preconditioned resid norm 1.180461366945e+03 true resid norm 2.955366363152e+00 ||r(i)||/||b|| 2.362188497832e-03 99 KSP Residual norm 1.123121972186e+03 99 KSP preconditioned resid norm 1.123121972186e+03 true resid norm 2.811881808413e+00 ||r(i)||/||b|| 2.247503033097e-03 100 KSP Residual norm 1.093372204175e+03 100 KSP preconditioned resid norm 1.093372204175e+03 true resid norm 2.765264562124e+00 ||r(i)||/||b|| 2.210242433411e-03 101 KSP Residual norm 1.051829589305e+03 101 KSP preconditioned resid norm 1.051829589305e+03 true resid norm 2.630402729486e+00 ||r(i)||/||b|| 2.102449005893e-03 102 KSP Residual norm 9.957500613922e+02 102 KSP preconditioned resid norm 9.957500613922e+02 true resid norm 2.653288559549e+00 ||r(i)||/||b|| 2.120741372353e-03 103 KSP Residual norm 9.909052701911e+02 103 KSP preconditioned resid norm 9.909052701911e+02 true resid norm 2.478755614431e+00 ||r(i)||/||b|| 1.981239305675e-03 104 KSP Residual norm 1.017112412332e+03 104 KSP preconditioned resid norm 1.017112412332e+03 true resid norm 2.588454519789e+00 ||r(i)||/||b|| 2.068920310539e-03 105 KSP Residual norm 9.164098176983e+02 105 KSP preconditioned resid norm 9.164098176983e+02 true resid norm 2.294176546256e+00 ||r(i)||/||b|| 1.833707494655e-03 106 KSP Residual norm 8.440470899369e+02 106 KSP preconditioned resid norm 8.440470899369e+02 true resid norm 2.983714803262e+00 ||r(i)||/||b|| 2.384847062265e-03 107 KSP Residual norm 7.760843214923e+02 107 KSP preconditioned resid norm 7.760843214923e+02 true resid norm 2.190507722450e+00 ||r(i)||/||b|| 1.750846260856e-03 108 KSP Residual norm 7.372578785432e+02 108 KSP preconditioned resid norm 7.372578785432e+02 true resid norm 1.846369710966e+00 ||r(i)||/||b|| 1.475780921232e-03 109 KSP Residual norm 5.351447337650e+02 109 KSP preconditioned resid norm 5.351447337650e+02 true resid norm 1.663741548026e+00 ||r(i)||/||b|| 1.329808445110e-03 110 KSP Residual norm 6.338563492236e+02 110 KSP preconditioned resid norm 6.338563492236e+02 true resid norm 1.593495944840e+00 ||r(i)||/||b|| 1.273661986269e-03 111 KSP Residual norm 5.965236157400e+02 111 KSP preconditioned resid norm 5.965236157400e+02 true resid norm 1.492048510785e+00 ||r(i)||/||b|| 1.192576282362e-03 112 KSP Residual norm 5.915101541003e+02 112 KSP preconditioned resid norm 5.915101541003e+02 true resid norm 1.852213218292e+00 ||r(i)||/||b|| 1.480451565780e-03 113 KSP Residual norm 4.674984103375e+02 113 KSP preconditioned resid norm 4.674984103375e+02 true resid norm 1.187371497777e+00 ||r(i)||/||b|| 9.490516403227e-04 114 KSP Residual norm 4.902618689281e+02 114 KSP preconditioned resid norm 4.902618689281e+02 true resid norm 1.273740654998e+00 ||r(i)||/||b|| 1.018085460393e-03 115 KSP Residual norm 4.221359689071e+02 115 KSP preconditioned resid norm 4.221359689071e+02 true resid norm 1.079476711698e+00 ||r(i)||/||b|| 8.628126461222e-04 116 KSP Residual norm 3.785709951339e+02 116 KSP preconditioned resid norm 3.785709951339e+02 true resid norm 9.500386039285e-01 ||r(i)||/||b|| 7.593543361247e-04 117 KSP Residual norm 3.475106274878e+02 117 KSP preconditioned resid norm 3.475106274878e+02 true resid norm 8.780824525144e-01 ||r(i)||/||b|| 7.018406568267e-04 118 KSP Residual norm 2.768119367980e+02 118 KSP preconditioned resid norm 2.768119367980e+02 true resid norm 6.955544944786e-01 ||r(i)||/||b|| 5.559482732695e-04 119 KSP Residual norm 2.471718512141e+02 119 KSP preconditioned resid norm 2.471718512141e+02 true resid norm 6.204771871084e-01 ||r(i)||/||b|| 4.959398918622e-04 120 KSP Residual norm 1.814609343744e+02 120 KSP preconditioned resid norm 1.814609343744e+02 true resid norm 4.575709374105e-01 ||r(i)||/||b|| 3.657309018502e-04 121 KSP Residual norm 1.544560577336e+02 121 KSP preconditioned resid norm 1.544560577336e+02 true resid norm 3.903117415851e-01 ||r(i)||/||b|| 3.119714421998e-04 122 KSP Residual norm 2.184994516704e+02 122 KSP preconditioned resid norm 2.184994516704e+02 true resid norm 5.482260955916e-01 ||r(i)||/||b|| 4.381904705164e-04 123 KSP Residual norm 1.534946334335e+02 123 KSP preconditioned resid norm 1.534946334335e+02 true resid norm 4.072508243598e-01 ||r(i)||/||b|| 3.255106456614e-04 124 KSP Residual norm 7.443810445294e+01 124 KSP preconditioned resid norm 7.443810445294e+01 true resid norm 1.963994901842e-01 ||r(i)||/||b|| 1.569797309998e-04 125 KSP Residual norm 7.683480526160e+01 125 KSP preconditioned resid norm 7.683480526160e+01 true resid norm 1.941426148544e-01 ||r(i)||/||b|| 1.551758379151e-04 126 KSP Residual norm 5.034604370666e+01 126 KSP preconditioned resid norm 5.034604370666e+01 true resid norm 1.300752077859e-01 ||r(i)||/||b|| 1.039675363150e-04 127 KSP Residual norm 4.137305218274e+01 127 KSP preconditioned resid norm 4.137305218274e+01 true resid norm 1.048014706704e-01 ||r(i)||/||b|| 8.376654470325e-05 128 KSP Residual norm 4.164622035780e+01 128 KSP preconditioned resid norm 4.164622035780e+01 true resid norm 1.075457849988e-01 ||r(i)||/||b|| 8.596004186895e-05 129 KSP Residual norm 7.384292012895e+01 129 KSP preconditioned resid norm 7.384292012895e+01 true resid norm 1.822883801112e-01 ||r(i)||/||b|| 1.457008918272e-04 130 KSP Residual norm 4.573030940300e+01 130 KSP preconditioned resid norm 4.573030940300e+01 true resid norm 1.286989100936e-01 ||r(i)||/||b|| 1.028674782582e-04 131 KSP Residual norm 3.163741418048e+01 131 KSP preconditioned resid norm 3.163741418048e+01 true resid norm 1.258572454385e-01 ||r(i)||/||b|| 1.005961701569e-04 132 KSP Residual norm 3.389871810924e+01 132 KSP preconditioned resid norm 3.389871810924e+01 true resid norm 2.793865673977e-01 ||r(i)||/||b|| 2.233102955303e-04 133 KSP Residual norm 2.860010279335e+01 133 KSP preconditioned resid norm 2.860010279335e+01 true resid norm 7.209670900329e-02 ||r(i)||/||b|| 5.762602527475e-05 134 KSP Residual norm 2.832432392362e+01 134 KSP preconditioned resid norm 2.832432392362e+01 true resid norm 7.248603269287e-02 ||r(i)||/||b|| 5.793720697897e-05 135 KSP Residual norm 2.718083022484e+01 135 KSP preconditioned resid norm 2.718083022484e+01 true resid norm 6.862410864615e-02 ||r(i)||/||b|| 5.485041791741e-05 136 KSP Residual norm 2.727163362677e+01 136 KSP preconditioned resid norm 2.727163362677e+01 true resid norm 6.966544463681e-02 ||r(i)||/||b|| 5.568274514769e-05 137 KSP Residual norm 2.713695922907e+01 137 KSP preconditioned resid norm 2.713695922907e+01 true resid norm 6.833133218399e-02 ||r(i)||/||b|| 5.461640524136e-05 138 KSP Residual norm 2.619925705214e+01 138 KSP preconditioned resid norm 2.619925705214e+01 true resid norm 6.568631538676e-02 ||r(i)||/||b|| 5.250227538832e-05 139 KSP Residual norm 2.597117927183e+01 139 KSP preconditioned resid norm 2.597117927183e+01 true resid norm 6.545796980242e-02 ||r(i)||/||b|| 5.231976153164e-05 140 KSP Residual norm 2.433347785527e+01 140 KSP preconditioned resid norm 2.433347785527e+01 true resid norm 6.107216729760e-02 ||r(i)||/||b|| 4.881424277098e-05 141 KSP Residual norm 2.520628368579e+01 141 KSP preconditioned resid norm 2.520628368579e+01 true resid norm 6.306107245129e-02 ||r(i)||/||b|| 5.040395054322e-05 142 KSP Residual norm 2.491075440536e+01 142 KSP preconditioned resid norm 2.491075440536e+01 true resid norm 6.226602431557e-02 ||r(i)||/||b|| 4.976847820895e-05 143 KSP Residual norm 2.428289420127e+01 143 KSP preconditioned resid norm 2.428289420127e+01 true resid norm 6.929350309462e-02 ||r(i)||/||b|| 5.538545678311e-05 144 KSP Residual norm 2.312338033265e+01 144 KSP preconditioned resid norm 2.312338033265e+01 true resid norm 5.876849943519e-02 ||r(i)||/||b|| 4.697294898242e-05 145 KSP Residual norm 2.302076234768e+01 145 KSP preconditioned resid norm 2.302076234768e+01 true resid norm 5.759867434546e-02 ||r(i)||/||b|| 4.603792197328e-05 146 KSP Residual norm 2.287088528237e+01 146 KSP preconditioned resid norm 2.287088528237e+01 true resid norm 7.178621996956e-02 ||r(i)||/||b|| 5.737785515502e-05 147 KSP Residual norm 2.209894945230e+01 147 KSP preconditioned resid norm 2.209894945230e+01 true resid norm 5.537821605452e-02 ||r(i)||/||b|| 4.426313658620e-05 148 KSP Residual norm 2.176867729430e+01 148 KSP preconditioned resid norm 2.176867729430e+01 true resid norm 5.602877688350e-02 ||r(i)||/||b|| 4.478312196822e-05 149 KSP Residual norm 2.186553533396e+01 149 KSP preconditioned resid norm 2.186553533396e+01 true resid norm 5.601449129641e-02 ||r(i)||/||b|| 4.477170367169e-05 150 KSP Residual norm 2.121485352395e+01 150 KSP preconditioned resid norm 2.121485352395e+01 true resid norm 5.317234796344e-02 ||r(i)||/||b|| 4.250001296896e-05 151 KSP Residual norm 2.087969621568e+01 151 KSP preconditioned resid norm 2.087969621568e+01 true resid norm 5.327159568932e-02 ||r(i)||/||b|| 4.257934047280e-05 152 KSP Residual norm 2.136179319415e+01 152 KSP preconditioned resid norm 2.136179319415e+01 true resid norm 1.216836200825e-01 ||r(i)||/||b|| 9.726024201849e-05 153 KSP Residual norm 2.015774474600e+01 153 KSP preconditioned resid norm 2.015774474600e+01 true resid norm 5.013310583192e-02 ||r(i)||/||b|| 4.007078358653e-05 154 KSP Residual norm 1.995683364220e+01 154 KSP preconditioned resid norm 1.995683364220e+01 true resid norm 5.229657683976e-02 ||r(i)||/||b|| 4.180001972924e-05 155 KSP Residual norm 2.009067877205e+01 155 KSP preconditioned resid norm 2.009067877205e+01 true resid norm 5.009526154611e-02 ||r(i)||/||b|| 4.004053510778e-05 156 KSP Residual norm 1.939838920774e+01 156 KSP preconditioned resid norm 1.939838920774e+01 true resid norm 4.813310721956e-02 ||r(i)||/||b|| 3.847220894729e-05 157 KSP Residual norm 1.816436830317e+01 157 KSP preconditioned resid norm 1.816436830317e+01 true resid norm 7.256172965874e-02 ||r(i)||/||b|| 5.799771064590e-05 158 KSP Residual norm 1.738513002130e+01 158 KSP preconditioned resid norm 1.738513002130e+01 true resid norm 5.612308573347e-02 ||r(i)||/||b|| 4.485850188844e-05 159 KSP Residual norm 1.679093370759e+01 159 KSP preconditioned resid norm 1.679093370759e+01 true resid norm 4.163169167526e-02 ||r(i)||/||b|| 3.327570633772e-05 160 KSP Residual norm 1.662589108018e+01 160 KSP preconditioned resid norm 1.662589108018e+01 true resid norm 4.125965648598e-02 ||r(i)||/||b|| 3.297834312215e-05 161 KSP Residual norm 1.626471494755e+01 161 KSP preconditioned resid norm 1.626471494755e+01 true resid norm 4.024242467199e-02 ||r(i)||/||b|| 3.216528206800e-05 162 KSP Residual norm 1.607123894212e+01 162 KSP preconditioned resid norm 1.607123894212e+01 true resid norm 3.971750073720e-02 ||r(i)||/||b|| 3.174571673206e-05 163 KSP Residual norm 1.659757764224e+01 163 KSP preconditioned resid norm 1.659757764224e+01 true resid norm 4.102229104953e-02 ||r(i)||/||b|| 3.278861980705e-05 164 KSP Residual norm 1.083952007078e+01 164 KSP preconditioned resid norm 1.083952007078e+01 true resid norm 7.445673422264e-02 ||r(i)||/||b|| 5.951236481534e-05 165 KSP Residual norm 8.294523584421e+00 165 KSP preconditioned resid norm 8.294523584421e+00 true resid norm 2.121851962126e-02 ||r(i)||/||b|| 1.695970544137e-05 166 KSP Residual norm 8.859352170399e+00 166 KSP preconditioned resid norm 8.859352170399e+00 true resid norm 2.667663915358e-02 ||r(i)||/||b|| 2.132231419938e-05 167 KSP Residual norm 8.660826535510e+00 167 KSP preconditioned resid norm 8.660826535510e+00 true resid norm 2.377403399532e-02 ||r(i)||/||b|| 1.900229709285e-05 168 KSP Residual norm 7.767745217146e+00 168 KSP preconditioned resid norm 7.767745217146e+00 true resid norm 1.922782789362e-02 ||r(i)||/||b|| 1.536856968222e-05 169 KSP Residual norm 7.680079615900e+00 169 KSP preconditioned resid norm 7.680079615900e+00 true resid norm 1.892247367237e-02 ||r(i)||/||b|| 1.512450375586e-05 170 KSP Residual norm 7.421902903029e+00 170 KSP preconditioned resid norm 7.421902903029e+00 true resid norm 1.817358491828e-02 ||r(i)||/||b|| 1.452592605559e-05 171 KSP Residual norm 7.339378675825e+00 171 KSP preconditioned resid norm 7.339378675825e+00 true resid norm 1.802896246421e-02 ||r(i)||/||b|| 1.441033108172e-05 172 KSP Residual norm 7.352688287357e+00 172 KSP preconditioned resid norm 7.352688287357e+00 true resid norm 1.821434604170e-02 ||r(i)||/||b|| 1.455850592729e-05 173 KSP Residual norm 6.850786362156e+00 173 KSP preconditioned resid norm 6.850786362156e+00 true resid norm 1.679036713879e-02 ||r(i)||/||b|| 1.342033685710e-05 174 KSP Residual norm 6.708564767905e+00 174 KSP preconditioned resid norm 6.708564767905e+00 true resid norm 1.844423427071e-02 ||r(i)||/||b|| 1.474225280115e-05 175 KSP Residual norm 7.115460685807e+00 175 KSP preconditioned resid norm 7.115460685807e+00 true resid norm 2.547469854595e-02 ||r(i)||/||b|| 2.036161764621e-05 176 KSP Residual norm 6.169792893619e+00 176 KSP preconditioned resid norm 6.169792893619e+00 true resid norm 1.520864223489e-02 ||r(i)||/||b|| 1.215608332112e-05 177 KSP Residual norm 6.103629956472e+00 177 KSP preconditioned resid norm 6.103629956472e+00 true resid norm 1.501672153499e-02 ||r(i)||/||b|| 1.200268343289e-05 178 KSP Residual norm 5.866688490937e+00 178 KSP preconditioned resid norm 5.866688490937e+00 true resid norm 1.439251676738e-02 ||r(i)||/||b|| 1.150376413112e-05 179 KSP Residual norm 5.785072117751e+00 179 KSP preconditioned resid norm 5.785072117751e+00 true resid norm 1.439123735410e-02 ||r(i)||/||b|| 1.150274151161e-05 180 KSP Residual norm 5.318019729873e+00 180 KSP preconditioned resid norm 5.318019729873e+00 true resid norm 1.314718553793e-02 ||r(i)||/||b|| 1.050838598009e-05 181 KSP Residual norm 5.068614830947e+00 181 KSP preconditioned resid norm 5.068614830947e+00 true resid norm 1.257194404693e-02 ||r(i)||/||b|| 1.004860243160e-05 182 KSP Residual norm 5.291037427847e+00 182 KSP preconditioned resid norm 5.291037427847e+00 true resid norm 1.794033717813e-02 ||r(i)||/||b|| 1.433949396521e-05 183 KSP Residual norm 5.137852126935e+00 183 KSP preconditioned resid norm 5.137852126935e+00 true resid norm 1.573597022688e-02 ||r(i)||/||b|| 1.257757019083e-05 184 KSP Residual norm 4.834342590953e+00 184 KSP preconditioned resid norm 4.834342590953e+00 true resid norm 1.199928409211e-02 ||r(i)||/||b|| 9.590882273686e-06 185 KSP Residual norm 4.827500421394e+00 185 KSP preconditioned resid norm 4.827500421394e+00 true resid norm 1.195789808418e-02 ||r(i)||/||b|| 9.557802939391e-06 186 KSP Residual norm 4.788027444960e+00 186 KSP preconditioned resid norm 4.788027444960e+00 true resid norm 1.178796284320e-02 ||r(i)||/||b|| 9.421975761873e-06 187 KSP Residual norm 4.547853280071e+00 187 KSP preconditioned resid norm 4.547853280071e+00 true resid norm 1.120283357801e-02 ||r(i)||/||b|| 8.954289035384e-06 188 KSP Residual norm 4.545440813785e+00 188 KSP preconditioned resid norm 4.545440813785e+00 true resid norm 1.117936434693e-02 ||r(i)||/||b|| 8.935530363565e-06 189 KSP Residual norm 5.155618737881e+00 189 KSP preconditioned resid norm 5.155618737881e+00 true resid norm 1.265446455221e-02 ||r(i)||/||b|| 1.011456007084e-05 190 KSP Residual norm 4.125611922039e+00 190 KSP preconditioned resid norm 4.125611922039e+00 true resid norm 1.159195237236e-02 ||r(i)||/||b|| 9.265306969325e-06 191 KSP Residual norm 4.031507742563e+00 191 KSP preconditioned resid norm 4.031507742563e+00 true resid norm 1.023118034099e-02 ||r(i)||/||b|| 8.177658385124e-06 192 KSP Residual norm 4.287079422280e+00 192 KSP preconditioned resid norm 4.287079422280e+00 true resid norm 1.274526101331e-02 ||r(i)||/||b|| 1.018713258122e-05 193 KSP Residual norm 3.738016626121e+00 193 KSP preconditioned resid norm 3.738016626121e+00 true resid norm 9.214822104030e-03 ||r(i)||/||b|| 7.365295570495e-06 194 KSP Residual norm 3.677813391005e+00 194 KSP preconditioned resid norm 3.677813391005e+00 true resid norm 9.243515387507e-03 ||r(i)||/||b|| 7.388229764048e-06 195 KSP Residual norm 3.525115925057e+00 195 KSP preconditioned resid norm 3.525115925057e+00 true resid norm 8.675466217658e-03 ||r(i)||/||b|| 6.934194950648e-06 196 KSP Residual norm 3.425306536141e+00 196 KSP preconditioned resid norm 3.425306536141e+00 true resid norm 8.544391129281e-03 ||r(i)||/||b|| 6.829428221902e-06 197 KSP Residual norm 3.322332260834e+00 197 KSP preconditioned resid norm 3.322332260834e+00 true resid norm 8.165477045569e-03 ||r(i)||/||b|| 6.526566789434e-06 198 KSP Residual norm 3.194730462398e+00 198 KSP preconditioned resid norm 3.194730462398e+00 true resid norm 7.867268146531e-03 ||r(i)||/||b|| 6.288212032460e-06 199 KSP Residual norm 3.106454365581e+00 199 KSP preconditioned resid norm 3.106454365581e+00 true resid norm 7.644406028886e-03 ||r(i)||/||b|| 6.110081044211e-06 200 KSP Residual norm 3.005681054907e+00 200 KSP preconditioned resid norm 3.005681054907e+00 true resid norm 7.379596756762e-03 ||r(i)||/||b|| 5.898422204032e-06 201 KSP Residual norm 2.894325320382e+00 201 KSP preconditioned resid norm 2.894325320382e+00 true resid norm 7.160282155903e-03 ||r(i)||/||b|| 5.723126702934e-06 202 KSP Residual norm 2.815118913538e+00 202 KSP preconditioned resid norm 2.815118913538e+00 true resid norm 6.907448056473e-03 ||r(i)||/||b|| 5.521039473080e-06 203 KSP Residual norm 2.699612161865e+00 203 KSP preconditioned resid norm 2.699612161865e+00 true resid norm 6.710821754890e-03 ||r(i)||/||b|| 5.363878454480e-06 204 KSP Residual norm 2.587306523820e+00 204 KSP preconditioned resid norm 2.587306523820e+00 true resid norm 6.347258022519e-03 ||r(i)||/||b|| 5.073286371703e-06 205 KSP Residual norm 2.493470795918e+00 205 KSP preconditioned resid norm 2.493470795918e+00 true resid norm 6.373941507309e-03 ||r(i)||/||b|| 5.094614157537e-06 206 KSP Residual norm 2.321086979769e+00 206 KSP preconditioned resid norm 2.321086979769e+00 true resid norm 5.692844768375e-03 ||r(i)||/||b|| 4.550221792962e-06 207 KSP Residual norm 2.234967245668e+00 207 KSP preconditioned resid norm 2.234967245668e+00 true resid norm 6.576294988882e-03 ||r(i)||/||b|| 5.256352841657e-06 1 SNES Function norm 6.576294999847e-03 0 KSP Residual norm 2.234967254054e+00 0 KSP preconditioned resid norm 2.234967254054e+00 true resid norm 6.576294999847e-03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 3.053301189831e+00 1 KSP preconditioned resid norm 3.053301189831e+00 true resid norm 2.204780592775e-02 ||r(i)||/||b|| 3.352618142626e+00 2 KSP Residual norm 7.505762542046e+00 2 KSP preconditioned resid norm 7.505762542046e+00 true resid norm 2.881652903481e-02 ||r(i)||/||b|| 4.381879011736e+00 3 KSP Residual norm 2.062019966315e+00 3 KSP preconditioned resid norm 2.062019966315e+00 true resid norm 2.397142547872e-02 ||r(i)||/||b|| 3.645126241946e+00 4 KSP Residual norm 1.370944375247e+00 4 KSP preconditioned resid norm 1.370944375247e+00 true resid norm 1.197115675609e-02 ||r(i)||/||b|| 1.820349719161e+00 5 KSP Residual norm 8.230060112790e-01 5 KSP preconditioned resid norm 8.230060112790e-01 true resid norm 3.541728277103e-03 ||r(i)||/||b|| 5.385598239107e-01 6 KSP Residual norm 7.773157265089e-01 6 KSP preconditioned resid norm 7.773157265089e-01 true resid norm 2.227631386672e-03 ||r(i)||/||b|| 3.387365358038e-01 7 KSP Residual norm 7.600849122572e-01 7 KSP preconditioned resid norm 7.600849122572e-01 true resid norm 2.142596620659e-03 ||r(i)||/||b|| 3.258060383102e-01 8 KSP Residual norm 7.390422962763e-01 8 KSP preconditioned resid norm 7.390422962763e-01 true resid norm 2.292784274191e-03 ||r(i)||/||b|| 3.486437689070e-01 9 KSP Residual norm 7.278413746333e-01 9 KSP preconditioned resid norm 7.278413746333e-01 true resid norm 1.788733931759e-03 ||r(i)||/||b|| 2.719972160313e-01 10 KSP Residual norm 7.249677475530e-01 10 KSP preconditioned resid norm 7.249677475530e-01 true resid norm 1.783544325903e-03 ||r(i)||/||b|| 2.712080777922e-01 11 KSP Residual norm 7.577023877258e-01 11 KSP preconditioned resid norm 7.577023877258e-01 true resid norm 1.880487907272e-03 ||r(i)||/||b|| 2.859494452904e-01 12 KSP Residual norm 7.251759729664e-01 12 KSP preconditioned resid norm 7.251759729664e-01 true resid norm 1.779402284213e-03 ||r(i)||/||b|| 2.705782335274e-01 13 KSP Residual norm 6.850946172844e-01 13 KSP preconditioned resid norm 6.850946172844e-01 true resid norm 1.707479098248e-03 ||r(i)||/||b|| 2.596415000069e-01 14 KSP Residual norm 6.730771379305e-01 14 KSP preconditioned resid norm 6.730771379305e-01 true resid norm 1.652121083640e-03 ||r(i)||/||b|| 2.512236880612e-01 15 KSP Residual norm 5.929718293202e-01 15 KSP preconditioned resid norm 5.929718293202e-01 true resid norm 1.453087183449e-03 ||r(i)||/||b|| 2.209583334510e-01 16 KSP Residual norm 5.556918557602e-01 16 KSP preconditioned resid norm 5.556918557602e-01 true resid norm 1.370441859834e-03 ||r(i)||/||b|| 2.083911776869e-01 17 KSP Residual norm 5.441060768266e-01 17 KSP preconditioned resid norm 5.441060768266e-01 true resid norm 1.324093932439e-03 ||r(i)||/||b|| 2.013434513612e-01 18 KSP Residual norm 3.199879367436e+00 18 KSP preconditioned resid norm 3.199879367436e+00 true resid norm 8.195037168800e-03 ||r(i)||/||b|| 1.246148046733e+00 19 KSP Residual norm 7.749658182913e-01 19 KSP preconditioned resid norm 7.749658182913e-01 true resid norm 1.897367984530e-03 ||r(i)||/||b|| 2.885162518673e-01 20 KSP Residual norm 8.078878658506e-01 20 KSP preconditioned resid norm 8.078878658506e-01 true resid norm 2.312254210905e-03 ||r(i)||/||b|| 3.516043928927e-01 21 KSP Residual norm 8.008391968277e-01 21 KSP preconditioned resid norm 8.008391968277e-01 true resid norm 1.968914069579e-03 ||r(i)||/||b|| 2.993956429303e-01 22 KSP Residual norm 7.968905907430e-01 22 KSP preconditioned resid norm 7.968905907430e-01 true resid norm 1.951682044953e-03 ||r(i)||/||b|| 2.967753187773e-01 23 KSP Residual norm 7.908028991773e-01 23 KSP preconditioned resid norm 7.908028991773e-01 true resid norm 2.126938983566e-03 ||r(i)||/||b|| 3.234251175800e-01 24 KSP Residual norm 7.994228619267e-01 24 KSP preconditioned resid norm 7.994228619267e-01 true resid norm 1.953713627466e-03 ||r(i)||/||b|| 2.970842438655e-01 25 KSP Residual norm 7.480780063843e-01 25 KSP preconditioned resid norm 7.480780063843e-01 true resid norm 2.185643984570e-03 ||r(i)||/||b|| 3.323518766450e-01 26 KSP Residual norm 7.365752449381e-01 26 KSP preconditioned resid norm 7.365752449381e-01 true resid norm 1.837270081382e-03 ||r(i)||/||b|| 2.793776862846e-01 27 KSP Residual norm 7.365702500494e-01 27 KSP preconditioned resid norm 7.365702500494e-01 true resid norm 1.809754844666e-03 ||r(i)||/||b|| 2.751936834811e-01 28 KSP Residual norm 7.365907941048e-01 28 KSP preconditioned resid norm 7.365907941048e-01 true resid norm 1.801366341985e-03 ||r(i)||/||b|| 2.739181168160e-01 29 KSP Residual norm 7.342896368831e-01 29 KSP preconditioned resid norm 7.342896368831e-01 true resid norm 1.835957197940e-03 ||r(i)||/||b|| 2.791780475149e-01 30 KSP Residual norm 7.296284953101e-01 30 KSP preconditioned resid norm 7.296284953101e-01 true resid norm 1.817522279409e-03 ||r(i)||/||b|| 2.763748097449e-01 31 KSP Residual norm 7.366138284036e-01 31 KSP preconditioned resid norm 7.366138284036e-01 true resid norm 1.801989228872e-03 ||r(i)||/||b|| 2.740128338089e-01 32 KSP Residual norm 6.822510293696e-01 32 KSP preconditioned resid norm 6.822510293696e-01 true resid norm 1.661219844069e-03 ||r(i)||/||b|| 2.526072574463e-01 33 KSP Residual norm 6.378081866025e-01 33 KSP preconditioned resid norm 6.378081866025e-01 true resid norm 1.588085292589e-03 ||r(i)||/||b|| 2.414863221048e-01 34 KSP Residual norm 5.600591648673e-01 34 KSP preconditioned resid norm 5.600591648673e-01 true resid norm 1.345736208033e-03 ||r(i)||/||b|| 2.046344040320e-01 35 KSP Residual norm 5.215609028535e-01 35 KSP preconditioned resid norm 5.215609028535e-01 true resid norm 1.249095393087e-03 ||r(i)||/||b|| 1.899390755913e-01 36 KSP Residual norm 5.270147807189e-01 36 KSP preconditioned resid norm 5.270147807189e-01 true resid norm 1.270603091905e-03 ||r(i)||/||b|| 1.932095643420e-01 37 KSP Residual norm 5.693285142250e-01 37 KSP preconditioned resid norm 5.693285142250e-01 true resid norm 1.371471027219e-03 ||r(i)||/||b|| 2.085476742225e-01 38 KSP Residual norm 5.915581598542e-01 38 KSP preconditioned resid norm 5.915581598542e-01 true resid norm 1.475922988905e-03 ||r(i)||/||b|| 2.244307758304e-01 39 KSP Residual norm 5.992375416469e-01 39 KSP preconditioned resid norm 5.992375416469e-01 true resid norm 1.472661165788e-03 ||r(i)||/||b|| 2.239347787503e-01 40 KSP Residual norm 5.993555983149e-01 40 KSP preconditioned resid norm 5.993555983149e-01 true resid norm 1.446150100360e-03 ||r(i)||/||b|| 2.199034715434e-01 41 KSP Residual norm 5.966227319018e-01 41 KSP preconditioned resid norm 5.966227319018e-01 true resid norm 1.453904764439e-03 ||r(i)||/||b|| 2.210826558834e-01 42 KSP Residual norm 5.914815507263e-01 42 KSP preconditioned resid norm 5.914815507263e-01 true resid norm 1.437315080296e-03 ||r(i)||/||b|| 2.185600068624e-01 43 KSP Residual norm 5.488004609008e-01 43 KSP preconditioned resid norm 5.488004609008e-01 true resid norm 2.116103630171e-03 ||r(i)||/||b|| 3.217774796022e-01 44 KSP Residual norm 4.966749277824e-01 44 KSP preconditioned resid norm 4.966749277824e-01 true resid norm 1.194817531762e-03 ||r(i)||/||b|| 1.816855131635e-01 45 KSP Residual norm 4.994223132666e-01 45 KSP preconditioned resid norm 4.994223132666e-01 true resid norm 1.859584017049e-03 ||r(i)||/||b|| 2.827707724626e-01 46 KSP Residual norm 4.787825848284e-01 46 KSP preconditioned resid norm 4.787825848284e-01 true resid norm 1.179422152121e-03 ||r(i)||/||b|| 1.793444716437e-01 47 KSP Residual norm 5.142615629465e-01 47 KSP preconditioned resid norm 5.142615629465e-01 true resid norm 1.253190695407e-03 ||r(i)||/||b|| 1.905618126066e-01 48 KSP Residual norm 5.020368883224e-01 48 KSP preconditioned resid norm 5.020368883224e-01 true resid norm 1.207291306159e-03 ||r(i)||/||b|| 1.835822915772e-01 49 KSP Residual norm 4.475420523794e-01 49 KSP preconditioned resid norm 4.475420523794e-01 true resid norm 2.228583752409e-03 ||r(i)||/||b|| 3.388813537807e-01 50 KSP Residual norm 4.019374852959e-01 50 KSP preconditioned resid norm 4.019374852959e-01 true resid norm 9.691000979312e-04 ||r(i)||/||b|| 1.473626256051e-01 51 KSP Residual norm 3.995126793507e-01 51 KSP preconditioned resid norm 3.995126793507e-01 true resid norm 1.003653810962e-03 ||r(i)||/||b|| 1.526169083026e-01 52 KSP Residual norm 3.968023152891e-01 52 KSP preconditioned resid norm 3.968023152891e-01 true resid norm 9.998969476897e-04 ||r(i)||/||b|| 1.520456347705e-01 53 KSP Residual norm 3.966305005480e-01 53 KSP preconditioned resid norm 3.966305005480e-01 true resid norm 9.854982362304e-04 ||r(i)||/||b|| 1.498561479151e-01 54 KSP Residual norm 3.915664626156e-01 54 KSP preconditioned resid norm 3.915664626156e-01 true resid norm 9.885838709726e-04 ||r(i)||/||b|| 1.503253535609e-01 55 KSP Residual norm 2.820616035180e-01 55 KSP preconditioned resid norm 2.820616035180e-01 true resid norm 7.054238416739e-04 ||r(i)||/||b|| 1.072676699708e-01 56 KSP Residual norm 1.632093530790e-01 56 KSP preconditioned resid norm 1.632093530790e-01 true resid norm 4.038056465348e-04 ||r(i)||/||b|| 6.140321359431e-02 57 KSP Residual norm 1.894025313104e-01 57 KSP preconditioned resid norm 1.894025313104e-01 true resid norm 4.613646195543e-04 ||r(i)||/||b|| 7.015570614838e-02 58 KSP Residual norm 1.670786826544e-01 58 KSP preconditioned resid norm 1.670786826544e-01 true resid norm 4.903635776097e-04 ||r(i)||/||b|| 7.456532555506e-02 59 KSP Residual norm 1.566365341480e-01 59 KSP preconditioned resid norm 1.566365341480e-01 true resid norm 4.351847008239e-04 ||r(i)||/||b|| 6.617475354040e-02 60 KSP Residual norm 2.024467462235e-01 60 KSP preconditioned resid norm 2.024467462235e-01 true resid norm 5.046102090099e-04 ||r(i)||/||b|| 7.673168691818e-02 61 KSP Residual norm 2.028859642417e-01 61 KSP preconditioned resid norm 2.028859642417e-01 true resid norm 5.695202291768e-04 ||r(i)||/||b|| 8.660198929490e-02 62 KSP Residual norm 1.177476456215e+00 62 KSP preconditioned resid norm 1.177476456215e+00 true resid norm 1.346428925389e-02 ||r(i)||/||b|| 2.047397395373e+00 63 KSP Residual norm 1.911566164492e-01 63 KSP preconditioned resid norm 1.911566164492e-01 true resid norm 4.835741637000e-04 ||r(i)||/||b|| 7.353291841550e-02 64 KSP Residual norm 1.775817887068e-01 64 KSP preconditioned resid norm 1.775817887068e-01 true resid norm 4.903203120383e-04 ||r(i)||/||b|| 7.455874653581e-02 65 KSP Residual norm 1.825198729780e-01 65 KSP preconditioned resid norm 1.825198729780e-01 true resid norm 5.344750061247e-04 ||r(i)||/||b|| 8.127296694219e-02 66 KSP Residual norm 1.799501128573e-01 66 KSP preconditioned resid norm 1.799501128573e-01 true resid norm 4.471350555039e-04 ||r(i)||/||b|| 6.799194006873e-02 67 KSP Residual norm 1.795429314981e-01 67 KSP preconditioned resid norm 1.795429314981e-01 true resid norm 4.543705604021e-04 ||r(i)||/||b|| 6.909218038618e-02 68 KSP Residual norm 1.778510873982e-01 68 KSP preconditioned resid norm 1.778510873982e-01 true resid norm 5.238696587190e-04 ||r(i)||/||b|| 7.966030397529e-02 69 KSP Residual norm 1.864278134963e-01 69 KSP preconditioned resid norm 1.864278134963e-01 true resid norm 4.615839582732e-04 ||r(i)||/||b|| 7.018905908022e-02 70 KSP Residual norm 1.841487663808e-01 70 KSP preconditioned resid norm 1.841487663808e-01 true resid norm 5.129635989703e-04 ||r(i)||/||b|| 7.800191429706e-02 71 KSP Residual norm 2.107697792557e-01 71 KSP preconditioned resid norm 2.107697792557e-01 true resid norm 5.377878336320e-04 ||r(i)||/||b|| 8.177671981633e-02 72 KSP Residual norm 6.918055454758e-01 72 KSP preconditioned resid norm 6.918055454758e-01 true resid norm 1.911424427661e-03 ||r(i)||/||b|| 2.906536929540e-01 73 KSP Residual norm 2.507537498152e-01 73 KSP preconditioned resid norm 2.507537498152e-01 true resid norm 6.536531489140e-04 ||r(i)||/||b|| 9.939535086689e-02 74 KSP Residual norm 1.898233509930e-01 74 KSP preconditioned resid norm 1.898233509930e-01 true resid norm 4.705586845766e-04 ||r(i)||/||b|| 7.155376767428e-02 75 KSP Residual norm 1.727574667829e-01 75 KSP preconditioned resid norm 1.727574667829e-01 true resid norm 4.529762892879e-04 ||r(i)||/||b|| 6.888016570096e-02 76 KSP Residual norm 1.721572263073e-01 76 KSP preconditioned resid norm 1.721572263073e-01 true resid norm 4.271595217321e-04 ||r(i)||/||b|| 6.495443433453e-02 77 KSP Residual norm 1.542921579198e-01 77 KSP preconditioned resid norm 1.542921579198e-01 true resid norm 7.286957010404e-04 ||r(i)||/||b|| 1.108064192767e-01 78 KSP Residual norm 1.452089631252e-01 78 KSP preconditioned resid norm 1.452089631252e-01 true resid norm 3.789653782714e-04 ||r(i)||/||b|| 5.762596998465e-02 79 KSP Residual norm 1.449137678694e-01 79 KSP preconditioned resid norm 1.449137678694e-01 true resid norm 3.604931750830e-04 ||r(i)||/||b|| 5.481706266087e-02 80 KSP Residual norm 1.438235866990e-01 80 KSP preconditioned resid norm 1.438235866990e-01 true resid norm 3.609994737848e-04 ||r(i)||/||b|| 5.489405110221e-02 81 KSP Residual norm 1.458089587083e-01 81 KSP preconditioned resid norm 1.458089587083e-01 true resid norm 3.652341195571e-04 ||r(i)||/||b|| 5.553797686473e-02 82 KSP Residual norm 1.947808194531e-01 82 KSP preconditioned resid norm 1.947808194531e-01 true resid norm 4.866720226118e-04 ||r(i)||/||b|| 7.400398288446e-02 83 KSP Residual norm 1.259092072656e-01 83 KSP preconditioned resid norm 1.259092072656e-01 true resid norm 3.127316969379e-04 ||r(i)||/||b|| 4.755438996353e-02 84 KSP Residual norm 1.255995072449e-01 84 KSP preconditioned resid norm 1.255995072449e-01 true resid norm 3.135240499674e-04 ||r(i)||/||b|| 4.767487619925e-02 85 KSP Residual norm 1.257011875115e-01 85 KSP preconditioned resid norm 1.257011875115e-01 true resid norm 3.132071747262e-04 ||r(i)||/||b|| 4.762669173652e-02 86 KSP Residual norm 1.266859091829e-01 86 KSP preconditioned resid norm 1.266859091829e-01 true resid norm 3.145620315748e-04 ||r(i)||/||b|| 4.783271303707e-02 87 KSP Residual norm 1.273156806528e-01 87 KSP preconditioned resid norm 1.273156806528e-01 true resid norm 3.180222574124e-04 ||r(i)||/||b|| 4.835887949366e-02 88 KSP Residual norm 1.261738666190e-01 88 KSP preconditioned resid norm 1.261738666190e-01 true resid norm 3.127308254865e-04 ||r(i)||/||b|| 4.755425744948e-02 89 KSP Residual norm 1.247690208452e-01 89 KSP preconditioned resid norm 1.247690208452e-01 true resid norm 3.472524929811e-04 ||r(i)||/||b|| 5.280366726085e-02 90 KSP Residual norm 1.687447401551e-01 90 KSP preconditioned resid norm 1.687447401551e-01 true resid norm 7.160840615427e-04 ||r(i)||/||b|| 1.088886769160e-01 91 KSP Residual norm 1.251209816585e-01 91 KSP preconditioned resid norm 1.251209816585e-01 true resid norm 3.168118505761e-04 ||r(i)||/||b|| 4.817482345051e-02 92 KSP Residual norm 1.207716989759e-01 92 KSP preconditioned resid norm 1.207716989759e-01 true resid norm 2.988661368207e-04 ||r(i)||/||b|| 4.544597479700e-02 93 KSP Residual norm 1.212272971096e-01 93 KSP preconditioned resid norm 1.212272971096e-01 true resid norm 3.015068784914e-04 ||r(i)||/||b|| 4.584752942172e-02 94 KSP Residual norm 1.198099738750e-01 94 KSP preconditioned resid norm 1.198099738750e-01 true resid norm 3.281682989592e-04 ||r(i)||/||b|| 4.990169981226e-02 95 KSP Residual norm 1.193534369316e-01 95 KSP preconditioned resid norm 1.193534369316e-01 true resid norm 3.010814265261e-04 ||r(i)||/||b|| 4.578283464064e-02 96 KSP Residual norm 1.185525858588e-01 96 KSP preconditioned resid norm 1.185525858588e-01 true resid norm 2.933691854372e-04 ||r(i)||/||b|| 4.461010119589e-02 97 KSP Residual norm 1.185803760794e-01 97 KSP preconditioned resid norm 1.185803760794e-01 true resid norm 2.932954828689e-04 ||r(i)||/||b|| 4.459889388717e-02 98 KSP Residual norm 1.185116805134e-01 98 KSP preconditioned resid norm 1.185116805134e-01 true resid norm 2.937227551280e-04 ||r(i)||/||b|| 4.466386546449e-02 99 KSP Residual norm 1.185064572177e-01 99 KSP preconditioned resid norm 1.185064572177e-01 true resid norm 2.953340511683e-04 ||r(i)||/||b|| 4.490888124319e-02 100 KSP Residual norm 1.182147182547e-01 100 KSP preconditioned resid norm 1.182147182547e-01 true resid norm 2.952062447532e-04 ||r(i)||/||b|| 4.488944683291e-02 101 KSP Residual norm 1.166140213338e-01 101 KSP preconditioned resid norm 1.166140213338e-01 true resid norm 2.883701324671e-04 ||r(i)||/||b|| 4.384993867730e-02 102 KSP Residual norm 1.155000171527e-01 102 KSP preconditioned resid norm 1.155000171527e-01 true resid norm 2.923877243934e-04 ||r(i)||/||b|| 4.446085894873e-02 103 KSP Residual norm 1.170637485508e-01 103 KSP preconditioned resid norm 1.170637485508e-01 true resid norm 3.021404913303e-04 ||r(i)||/||b|| 4.594387741689e-02 104 KSP Residual norm 9.401192845136e-02 104 KSP preconditioned resid norm 9.401192845136e-02 true resid norm 3.508009926812e-04 ||r(i)||/||b|| 5.334325675618e-02 105 KSP Residual norm 1.025945929185e-01 105 KSP preconditioned resid norm 1.025945929185e-01 true resid norm 2.531639779640e-04 ||r(i)||/||b|| 3.849644487813e-02 106 KSP Residual norm 9.971857287842e-02 106 KSP preconditioned resid norm 9.971857287842e-02 true resid norm 2.484024336819e-04 ||r(i)||/||b|| 3.777239824061e-02 107 KSP Residual norm 9.796480061327e-02 107 KSP preconditioned resid norm 9.796480061327e-02 true resid norm 2.458800289609e-04 ||r(i)||/||b|| 3.738883808689e-02 108 KSP Residual norm 8.201533808992e-02 108 KSP preconditioned resid norm 8.201533808992e-02 true resid norm 2.089697465787e-04 ||r(i)||/||b|| 3.177621237848e-02 109 KSP Residual norm 8.171693273263e-02 109 KSP preconditioned resid norm 8.171693273263e-02 true resid norm 1.999408296649e-04 ||r(i)||/||b|| 3.040326348948e-02 110 KSP Residual norm 8.521103311896e-02 110 KSP preconditioned resid norm 8.521103311896e-02 true resid norm 2.098140162597e-04 ||r(i)||/||b|| 3.190459312799e-02 111 KSP Residual norm 8.608747083078e-02 111 KSP preconditioned resid norm 8.608747083078e-02 true resid norm 2.114258936799e-04 ||r(i)||/||b|| 3.214969731207e-02 112 KSP Residual norm 8.609448973241e-02 112 KSP preconditioned resid norm 8.609448973241e-02 true resid norm 2.113025760812e-04 ||r(i)||/||b|| 3.213094547707e-02 113 KSP Residual norm 8.607923285280e-02 113 KSP preconditioned resid norm 8.607923285280e-02 true resid norm 2.111539607078e-04 ||r(i)||/||b|| 3.210834682944e-02 114 KSP Residual norm 8.628477993968e-02 114 KSP preconditioned resid norm 8.628477993968e-02 true resid norm 2.116352848455e-04 ||r(i)||/||b|| 3.218153760596e-02 115 KSP Residual norm 8.656124183445e-02 115 KSP preconditioned resid norm 8.656124183445e-02 true resid norm 2.126899818585e-04 ||r(i)||/||b|| 3.234191621018e-02 116 KSP Residual norm 8.963438283064e-02 116 KSP preconditioned resid norm 8.963438283064e-02 true resid norm 2.200222878116e-04 ||r(i)||/||b|| 3.345687622236e-02 117 KSP Residual norm 8.922939745312e-02 117 KSP preconditioned resid norm 8.922939745312e-02 true resid norm 2.200712707285e-04 ||r(i)||/||b|| 3.346432462863e-02 118 KSP Residual norm 8.946182573087e-02 118 KSP preconditioned resid norm 8.946182573087e-02 true resid norm 2.195556312929e-04 ||r(i)||/||b|| 3.338591582312e-02 119 KSP Residual norm 9.216846295679e-02 119 KSP preconditioned resid norm 9.216846295679e-02 true resid norm 2.374763390575e-04 ||r(i)||/||b|| 3.611096203303e-02 120 KSP Residual norm 9.100545280281e-02 120 KSP preconditioned resid norm 9.100545280281e-02 true resid norm 2.234591614345e-04 ||r(i)||/||b|| 3.397949171072e-02 121 KSP Residual norm 8.622843110599e-02 121 KSP preconditioned resid norm 8.622843110599e-02 true resid norm 2.779751738722e-04 ||r(i)||/||b|| 4.226926770753e-02 122 KSP Residual norm 7.851556192614e-02 122 KSP preconditioned resid norm 7.851556192614e-02 true resid norm 1.980290466742e-04 ||r(i)||/||b|| 3.011255527296e-02 123 KSP Residual norm 7.329170132471e-02 123 KSP preconditioned resid norm 7.329170132471e-02 true resid norm 1.799022417457e-04 ||r(i)||/||b|| 2.735616965934e-02 124 KSP Residual norm 7.191464961955e-02 124 KSP preconditioned resid norm 7.191464961955e-02 true resid norm 1.763521259665e-04 ||r(i)||/||b|| 2.681633442091e-02 125 KSP Residual norm 7.276320048823e-02 125 KSP preconditioned resid norm 7.276320048823e-02 true resid norm 1.789120848430e-04 ||r(i)||/||b|| 2.720560510852e-02 126 KSP Residual norm 6.596607907958e-02 126 KSP preconditioned resid norm 6.596607907958e-02 true resid norm 1.615772171752e-04 ||r(i)||/||b|| 2.456964250828e-02 127 KSP Residual norm 6.587724532020e-02 127 KSP preconditioned resid norm 6.587724532020e-02 true resid norm 1.670553199735e-04 ||r(i)||/||b|| 2.540264996893e-02 128 KSP Residual norm 6.991439835553e-02 128 KSP preconditioned resid norm 6.991439835553e-02 true resid norm 2.171341354040e-04 ||r(i)||/||b|| 3.301769999811e-02 129 KSP Residual norm 6.559303770183e-02 129 KSP preconditioned resid norm 6.559303770183e-02 true resid norm 1.607357866677e-04 ||r(i)||/||b|| 2.444169348721e-02 130 KSP Residual norm 6.547102661426e-02 130 KSP preconditioned resid norm 6.547102661426e-02 true resid norm 1.611657554670e-04 ||r(i)||/||b|| 2.450707510395e-02 131 KSP Residual norm 6.636865557514e-02 131 KSP preconditioned resid norm 6.636865557514e-02 true resid norm 1.715609839397e-04 ||r(i)||/||b|| 2.608778711169e-02 132 KSP Residual norm 6.595691031344e-02 132 KSP preconditioned resid norm 6.595691031344e-02 true resid norm 1.663867285301e-04 ||r(i)||/||b|| 2.530098308150e-02 133 KSP Residual norm 6.319302210473e-02 133 KSP preconditioned resid norm 6.319302210473e-02 true resid norm 1.599490805640e-04 ||r(i)||/||b|| 2.432206593040e-02 134 KSP Residual norm 6.169101977153e-02 134 KSP preconditioned resid norm 6.169101977153e-02 true resid norm 1.515739299060e-04 ||r(i)||/||b|| 2.304852959143e-02 135 KSP Residual norm 5.786167019170e-02 135 KSP preconditioned resid norm 5.786167019170e-02 true resid norm 1.426145972346e-04 ||r(i)||/||b|| 2.168616177314e-02 136 KSP Residual norm 1.953996353345e-01 136 KSP preconditioned resid norm 1.953996353345e-01 true resid norm 4.784038636328e-04 ||r(i)||/||b|| 7.274671583983e-02 137 KSP Residual norm 9.785094004570e-02 137 KSP preconditioned resid norm 9.785094004570e-02 true resid norm 2.685376886960e-04 ||r(i)||/||b|| 4.083419139535e-02 138 KSP Residual norm 8.385165564328e-02 138 KSP preconditioned resid norm 8.385165564328e-02 true resid norm 2.064817349679e-04 ||r(i)||/||b|| 3.139788208600e-02 139 KSP Residual norm 8.981048692029e-02 139 KSP preconditioned resid norm 8.981048692029e-02 true resid norm 2.205664646087e-04 ||r(i)||/||b|| 3.353962445630e-02 140 KSP Residual norm 8.928449798890e-02 140 KSP preconditioned resid norm 8.928449798890e-02 true resid norm 2.186203183152e-04 ||r(i)||/||b|| 3.324369091112e-02 141 KSP Residual norm 1.015213545922e-01 141 KSP preconditioned resid norm 1.015213545922e-01 true resid norm 2.489685698526e-04 ||r(i)||/||b|| 3.785848564554e-02 142 KSP Residual norm 5.230454543653e-02 142 KSP preconditioned resid norm 5.230454543653e-02 true resid norm 1.283151817532e-04 ||r(i)||/||b|| 1.951177399375e-02 143 KSP Residual norm 5.172817643254e-02 143 KSP preconditioned resid norm 5.172817643254e-02 true resid norm 1.271593656505e-04 ||r(i)||/||b|| 1.933601908878e-02 144 KSP Residual norm 1.090244692132e-01 144 KSP preconditioned resid norm 1.090244692132e-01 true resid norm 2.877099013394e-04 ||r(i)||/||b|| 4.374954307039e-02 145 KSP Residual norm 5.139312083957e-02 145 KSP preconditioned resid norm 5.139312083957e-02 true resid norm 1.336773820321e-04 ||r(i)||/||b|| 2.032715716604e-02 146 KSP Residual norm 5.053521310632e-02 146 KSP preconditioned resid norm 5.053521310632e-02 true resid norm 1.253607350137e-04 ||r(i)||/||b|| 1.906251696686e-02 147 KSP Residual norm 5.050215516671e-02 147 KSP preconditioned resid norm 5.050215516671e-02 true resid norm 1.240840006327e-04 ||r(i)||/||b|| 1.886837507070e-02 148 KSP Residual norm 4.998644131400e-02 148 KSP preconditioned resid norm 4.998644131400e-02 true resid norm 1.230467130276e-04 ||r(i)||/||b|| 1.871064376378e-02 149 KSP Residual norm 4.972056932271e-02 149 KSP preconditioned resid norm 4.972056932271e-02 true resid norm 1.219370787335e-04 ||r(i)||/||b|| 1.854191132489e-02 150 KSP Residual norm 4.965742587362e-02 150 KSP preconditioned resid norm 4.965742587362e-02 true resid norm 1.218484779060e-04 ||r(i)||/||b|| 1.852843856743e-02 151 KSP Residual norm 4.971836161329e-02 151 KSP preconditioned resid norm 4.971836161329e-02 true resid norm 1.218868764038e-04 ||r(i)||/||b|| 1.853427749313e-02 152 KSP Residual norm 4.859007686697e-02 152 KSP preconditioned resid norm 4.859007686697e-02 true resid norm 1.190101418449e-04 ||r(i)||/||b|| 1.809683748185e-02 153 KSP Residual norm 4.854631070200e-02 153 KSP preconditioned resid norm 4.854631070200e-02 true resid norm 1.207436796998e-04 ||r(i)||/||b|| 1.836044151040e-02 154 KSP Residual norm 4.832030185138e-02 154 KSP preconditioned resid norm 4.832030185138e-02 true resid norm 1.183314264960e-04 ||r(i)||/||b|| 1.799363114013e-02 155 KSP Residual norm 4.766620630012e-02 155 KSP preconditioned resid norm 4.766620630012e-02 true resid norm 1.169587059958e-04 ||r(i)||/||b|| 1.778489346943e-02 156 KSP Residual norm 4.778726716958e-02 156 KSP preconditioned resid norm 4.778726716958e-02 true resid norm 1.173330484098e-04 ||r(i)||/||b|| 1.784181646544e-02 157 KSP Residual norm 4.692203602826e-02 157 KSP preconditioned resid norm 4.692203602826e-02 true resid norm 1.149657787596e-04 ||r(i)||/||b|| 1.748184635305e-02 158 KSP Residual norm 4.691153582306e-02 158 KSP preconditioned resid norm 4.691153582306e-02 true resid norm 1.148520428232e-04 ||r(i)||/||b|| 1.746455151812e-02 159 KSP Residual norm 4.682043522042e-02 159 KSP preconditioned resid norm 4.682043522042e-02 true resid norm 1.254064611552e-04 ||r(i)||/||b|| 1.906947014361e-02 160 KSP Residual norm 4.652420233440e-02 160 KSP preconditioned resid norm 4.652420233440e-02 true resid norm 1.138670531095e-04 ||r(i)||/||b|| 1.731477269680e-02 161 KSP Residual norm 4.635770744010e-02 161 KSP preconditioned resid norm 4.635770744010e-02 true resid norm 1.135794338392e-04 ||r(i)||/||b|| 1.727103693521e-02 162 KSP Residual norm 5.056432300688e-02 162 KSP preconditioned resid norm 5.056432300688e-02 true resid norm 1.239050388236e-04 ||r(i)||/||b|| 1.884116190446e-02 163 KSP Residual norm 4.694087492725e-02 163 KSP preconditioned resid norm 4.694087492725e-02 true resid norm 1.169964323223e-04 ||r(i)||/||b|| 1.779063018386e-02 164 KSP Residual norm 4.548481298938e-02 164 KSP preconditioned resid norm 4.548481298938e-02 true resid norm 1.113279757590e-04 ||r(i)||/||b|| 1.692867728130e-02 165 KSP Residual norm 4.529109045483e-02 165 KSP preconditioned resid norm 4.529109045483e-02 true resid norm 1.108995286484e-04 ||r(i)||/||b|| 1.686352705452e-02 166 KSP Residual norm 4.378761586850e-02 166 KSP preconditioned resid norm 4.378761586850e-02 true resid norm 1.070349230319e-04 ||r(i)||/||b|| 1.627587008101e-02 167 KSP Residual norm 3.771815909111e-02 167 KSP preconditioned resid norm 3.771815909111e-02 true resid norm 9.197136529824e-05 ||r(i)||/||b|| 1.398528583349e-02 168 KSP Residual norm 3.787748803506e-02 168 KSP preconditioned resid norm 3.787748803506e-02 true resid norm 9.232367270365e-05 ||r(i)||/||b|| 1.403885815734e-02 169 KSP Residual norm 3.392764903681e-02 169 KSP preconditioned resid norm 3.392764903681e-02 true resid norm 9.079561359189e-05 ||r(i)||/||b|| 1.380649949462e-02 170 KSP Residual norm 3.353015297083e-02 170 KSP preconditioned resid norm 3.353015297083e-02 true resid norm 8.214148228984e-05 ||r(i)||/||b|| 1.249054099485e-02 171 KSP Residual norm 3.964061234082e-02 171 KSP preconditioned resid norm 3.964061234082e-02 true resid norm 9.675004117346e-05 ||r(i)||/||b|| 1.471193752344e-02 172 KSP Residual norm 3.783078217673e-02 172 KSP preconditioned resid norm 3.783078217673e-02 true resid norm 9.885811535208e-05 ||r(i)||/||b|| 1.503249403416e-02 173 KSP Residual norm 3.703371486770e-02 173 KSP preconditioned resid norm 3.703371486770e-02 true resid norm 9.249365695967e-05 ||r(i)||/||b|| 1.406470618514e-02 174 KSP Residual norm 3.816193241235e-02 174 KSP preconditioned resid norm 3.816193241235e-02 true resid norm 9.438951497491e-05 ||r(i)||/||b|| 1.435299282911e-02 175 KSP Residual norm 3.784529698307e-02 175 KSP preconditioned resid norm 3.784529698307e-02 true resid norm 9.743151555500e-05 ||r(i)||/||b|| 1.481556340725e-02 176 KSP Residual norm 3.796802084705e-02 176 KSP preconditioned resid norm 3.796802084705e-02 true resid norm 9.274124494396e-05 ||r(i)||/||b|| 1.410235473715e-02 177 KSP Residual norm 3.913463167687e-02 177 KSP preconditioned resid norm 3.913463167687e-02 true resid norm 9.546159940985e-05 ||r(i)||/||b|| 1.451601538740e-02 178 KSP Residual norm 3.855310645398e-02 178 KSP preconditioned resid norm 3.855310645398e-02 true resid norm 1.039305646355e-04 ||r(i)||/||b|| 1.580381729194e-02 179 KSP Residual norm 3.826518088121e-02 179 KSP preconditioned resid norm 3.826518088121e-02 true resid norm 9.333620057331e-05 ||r(i)||/||b|| 1.419282446658e-02 180 KSP Residual norm 3.792000245285e-02 180 KSP preconditioned resid norm 3.792000245285e-02 true resid norm 9.253234843888e-05 ||r(i)||/||b|| 1.407058966196e-02 181 KSP Residual norm 3.907075424613e-02 181 KSP preconditioned resid norm 3.907075424613e-02 true resid norm 9.537382049433e-05 ||r(i)||/||b|| 1.450266761095e-02 182 KSP Residual norm 2.251970168164e-02 182 KSP preconditioned resid norm 2.251970168164e-02 true resid norm 5.508754346601e-05 ||r(i)||/||b|| 8.376683750850e-03 183 KSP Residual norm 2.618759419196e-02 183 KSP preconditioned resid norm 2.618759419196e-02 true resid norm 6.411467333578e-05 ||r(i)||/||b|| 9.749360899606e-03 184 KSP Residual norm 2.495952378536e-02 184 KSP preconditioned resid norm 2.495952378536e-02 true resid norm 7.773709375516e-05 ||r(i)||/||b|| 1.182080392637e-02 185 KSP Residual norm 2.956686311283e-02 185 KSP preconditioned resid norm 2.956686311283e-02 true resid norm 7.255857271215e-05 ||r(i)||/||b|| 1.103335125840e-02 186 KSP Residual norm 2.936012064397e-02 186 KSP preconditioned resid norm 2.936012064397e-02 true resid norm 7.150981171005e-05 ||r(i)||/||b|| 1.087387529174e-02 187 KSP Residual norm 2.949165926565e-02 187 KSP preconditioned resid norm 2.949165926565e-02 true resid norm 7.164347089992e-05 ||r(i)||/||b|| 1.089419968259e-02 188 KSP Residual norm 2.899187821530e-02 188 KSP preconditioned resid norm 2.899187821530e-02 true resid norm 8.814901877057e-05 ||r(i)||/||b|| 1.340405483218e-02 189 KSP Residual norm 2.892997483292e-02 189 KSP preconditioned resid norm 2.892997483292e-02 true resid norm 8.689633924612e-05 ||r(i)||/||b|| 1.321357074890e-02 190 KSP Residual norm 2.893996314782e-02 190 KSP preconditioned resid norm 2.893996314782e-02 true resid norm 7.127851200055e-05 ||r(i)||/||b|| 1.083870355606e-02 191 KSP Residual norm 2.828683206047e-02 191 KSP preconditioned resid norm 2.828683206047e-02 true resid norm 7.199001232194e-05 ||r(i)||/||b|| 1.094689522347e-02 192 KSP Residual norm 3.122298489618e-02 192 KSP preconditioned resid norm 3.122298489618e-02 true resid norm 7.596880568965e-05 ||r(i)||/||b|| 1.155191573544e-02 193 KSP Residual norm 1.861923809752e-02 193 KSP preconditioned resid norm 1.861923809752e-02 true resid norm 4.619470286525e-05 ||r(i)||/||b|| 7.024426803591e-03 194 KSP Residual norm 2.134900808030e-02 194 KSP preconditioned resid norm 2.134900808030e-02 true resid norm 5.174247491301e-05 ||r(i)||/||b|| 7.868028261234e-03 195 KSP Residual norm 1.965874729318e-02 195 KSP preconditioned resid norm 1.965874729318e-02 true resid norm 4.778805845790e-05 ||r(i)||/||b|| 7.266714534402e-03 196 KSP Residual norm 1.962511115208e-02 196 KSP preconditioned resid norm 1.962511115208e-02 true resid norm 4.758335798344e-05 ||r(i)||/||b|| 7.235587513113e-03 197 KSP Residual norm 1.669956839334e-02 197 KSP preconditioned resid norm 1.669956839334e-02 true resid norm 4.038321820274e-05 ||r(i)||/||b|| 6.140724861595e-03 198 KSP Residual norm 1.343377833659e-02 198 KSP preconditioned resid norm 1.343377833659e-02 true resid norm 6.278792496025e-05 ||r(i)||/||b|| 9.547613810164e-03 199 KSP Residual norm 1.220560590198e-02 199 KSP preconditioned resid norm 1.220560590198e-02 true resid norm 3.068525904682e-05 ||r(i)||/||b|| 4.666040536127e-03 200 KSP Residual norm 1.189969852569e-02 200 KSP preconditioned resid norm 1.189969852569e-02 true resid norm 4.812776416378e-05 ||r(i)||/||b|| 7.318370627366e-03 201 KSP Residual norm 9.862814898502e-03 201 KSP preconditioned resid norm 9.862814898502e-03 true resid norm 6.469095353796e-05 ||r(i)||/||b|| 9.836990819218e-03 202 KSP Residual norm 1.003996149403e-02 202 KSP preconditioned resid norm 1.003996149403e-02 true resid norm 2.473184865881e-05 ||r(i)||/||b|| 3.760757183092e-03 203 KSP Residual norm 9.848454398197e-03 203 KSP preconditioned resid norm 9.848454398197e-03 true resid norm 2.465178023752e-05 ||r(i)||/||b|| 3.748581874459e-03 204 KSP Residual norm 9.740860200636e-03 204 KSP preconditioned resid norm 9.740860200636e-03 true resid norm 2.398785321700e-05 ||r(i)||/||b|| 3.647624265268e-03 205 KSP Residual norm 9.660640889815e-03 205 KSP preconditioned resid norm 9.660640889815e-03 true resid norm 2.447565981725e-05 ||r(i)||/||b|| 3.721800773508e-03 206 KSP Residual norm 1.554420455183e-02 206 KSP preconditioned resid norm 1.554420455183e-02 true resid norm 4.647386628436e-05 ||r(i)||/||b|| 7.066876757420e-03 207 KSP Residual norm 1.969007714491e-02 207 KSP preconditioned resid norm 1.969007714491e-02 true resid norm 6.738363916125e-05 ||r(i)||/||b|| 1.024644410915e-02 208 KSP Residual norm 1.151367107474e-02 208 KSP preconditioned resid norm 1.151367107474e-02 true resid norm 2.871318380918e-05 ||r(i)||/||b|| 4.366164201857e-03 209 KSP Residual norm 7.405054954702e-03 209 KSP preconditioned resid norm 7.405054954702e-03 true resid norm 1.836277021098e-05 ||r(i)||/||b|| 2.792266802417e-03 210 KSP Residual norm 8.205759829359e-03 210 KSP preconditioned resid norm 8.205759829359e-03 true resid norm 2.036627609945e-05 ||r(i)||/||b|| 3.096922522473e-03 211 KSP Residual norm 7.379765435698e-03 211 KSP preconditioned resid norm 7.379765435698e-03 true resid norm 1.875920133823e-05 ||r(i)||/||b|| 2.852548636985e-03 212 KSP Residual norm 4.040977584858e-03 212 KSP preconditioned resid norm 4.040977584858e-03 true resid norm 1.221614692269e-05 ||r(i)||/||b|| 1.857603243615e-03 213 KSP Residual norm 1.927082753675e-02 213 KSP preconditioned resid norm 1.927082753675e-02 true resid norm 4.763125305830e-05 ||r(i)||/||b|| 7.242870500701e-03 214 KSP Residual norm 1.560387165308e-02 214 KSP preconditioned resid norm 1.560387165308e-02 true resid norm 3.838776247830e-05 ||r(i)||/||b|| 5.837293259989e-03 215 KSP Residual norm 1.557378997808e-02 215 KSP preconditioned resid norm 1.557378997808e-02 true resid norm 3.839948069835e-05 ||r(i)||/||b|| 5.839075147822e-03 216 KSP Residual norm 1.469516174825e-02 216 KSP preconditioned resid norm 1.469516174825e-02 true resid norm 3.608452414917e-05 ||r(i)||/||b|| 5.487059833844e-03 217 KSP Residual norm 5.842813997490e-03 217 KSP preconditioned resid norm 5.842813997490e-03 true resid norm 1.448847916253e-05 ||r(i)||/||b|| 2.203137049489e-03 218 KSP Residual norm 6.569185702633e-03 218 KSP preconditioned resid norm 6.569185702633e-03 true resid norm 1.690405410702e-05 ||r(i)||/||b|| 2.570452527968e-03 219 KSP Residual norm 4.613813163559e-03 219 KSP preconditioned resid norm 4.613813163559e-03 true resid norm 1.146874770839e-05 ||r(i)||/||b|| 1.743952743704e-03 220 KSP Residual norm 3.896184935869e-03 220 KSP preconditioned resid norm 3.896184935869e-03 true resid norm 9.798244504010e-06 ||r(i)||/||b|| 1.489933846374e-03 221 KSP Residual norm 3.736568542452e-03 221 KSP preconditioned resid norm 3.736568542452e-03 true resid norm 9.326246289972e-06 ||r(i)||/||b|| 1.418161181971e-03 222 KSP Residual norm 3.324752246104e-03 222 KSP preconditioned resid norm 3.324752246104e-03 true resid norm 1.072849586940e-05 ||r(i)||/||b|| 1.631389083009e-03 223 KSP Residual norm 3.260096629858e-03 223 KSP preconditioned resid norm 3.260096629858e-03 true resid norm 9.811533585421e-06 ||r(i)||/||b|| 1.491954601436e-03 224 KSP Residual norm 3.173067604491e-03 224 KSP preconditioned resid norm 3.173067604491e-03 true resid norm 9.456837306268e-06 ||r(i)||/||b|| 1.438019022335e-03 225 KSP Residual norm 2.972671140395e-03 225 KSP preconditioned resid norm 2.972671140395e-03 true resid norm 7.862107273887e-06 ||r(i)||/||b|| 1.195522292426e-03 226 KSP Residual norm 2.959015926316e-03 226 KSP preconditioned resid norm 2.959015926316e-03 true resid norm 7.320659804563e-06 ||r(i)||/||b|| 1.113189083630e-03 227 KSP Residual norm 2.973939981092e-03 227 KSP preconditioned resid norm 2.973939981092e-03 true resid norm 7.329737240941e-06 ||r(i)||/||b|| 1.114569410452e-03 228 KSP Residual norm 2.738298396538e-03 228 KSP preconditioned resid norm 2.738298396538e-03 true resid norm 6.759724728922e-06 ||r(i)||/||b|| 1.027892563986e-03 229 KSP Residual norm 7.749502692089e-03 229 KSP preconditioned resid norm 7.749502692089e-03 true resid norm 1.902650360004e-05 ||r(i)||/||b|| 2.893194967756e-03 230 KSP Residual norm 1.043591752635e-03 230 KSP preconditioned resid norm 1.043591752635e-03 true resid norm 2.577161682950e-06 ||r(i)||/||b|| 3.918865688066e-04 231 KSP Residual norm 8.366913744158e-03 231 KSP preconditioned resid norm 8.366913744158e-03 true resid norm 2.045402294302e-05 ||r(i)||/||b|| 3.110265422019e-03 232 KSP Residual norm 5.708199253729e-03 232 KSP preconditioned resid norm 5.708199253729e-03 true resid norm 1.393843240282e-05 ||r(i)||/||b|| 2.119496221374e-03 233 KSP Residual norm 5.942071519770e-03 233 KSP preconditioned resid norm 5.942071519770e-03 true resid norm 1.486785296443e-05 ||r(i)||/||b|| 2.260825124903e-03 234 KSP Residual norm 5.443535892602e-03 234 KSP preconditioned resid norm 5.443535892602e-03 true resid norm 1.336949655473e-05 ||r(i)||/||b|| 2.032983093830e-03 235 KSP Residual norm 5.247554066555e-03 235 KSP preconditioned resid norm 5.247554066555e-03 true resid norm 1.289891118924e-05 ||r(i)||/||b|| 1.961425269022e-03 236 KSP Residual norm 5.359556709823e-03 236 KSP preconditioned resid norm 5.359556709823e-03 true resid norm 1.326813439378e-05 ||r(i)||/||b|| 2.017569831354e-03 237 KSP Residual norm 5.735412250342e-03 237 KSP preconditioned resid norm 5.735412250342e-03 true resid norm 1.412480827360e-05 ||r(i)||/||b|| 2.147836779513e-03 238 KSP Residual norm 1.229050507061e-01 238 KSP preconditioned resid norm 1.229050507061e-01 true resid norm 3.011508180563e-04 ||r(i)||/||b|| 4.579338640728e-02 239 KSP Residual norm 2.529018392250e-02 239 KSP preconditioned resid norm 2.529018392250e-02 true resid norm 6.202401285850e-05 ||r(i)||/||b|| 9.431452339036e-03 240 KSP Residual norm 1.909133071042e-02 240 KSP preconditioned resid norm 1.909133071042e-02 true resid norm 4.681112535005e-05 ||r(i)||/||b|| 7.118160811086e-03 241 KSP Residual norm 6.570050114857e-02 241 KSP preconditioned resid norm 6.570050114857e-02 true resid norm 1.614797812145e-04 ||r(i)||/||b|| 2.455482626893e-02 242 KSP Residual norm 2.918336493687e-03 242 KSP preconditioned resid norm 2.918336493687e-03 true resid norm 7.225238361721e-06 ||r(i)||/||b|| 1.098679174503e-03 243 KSP Residual norm 3.878468605244e-03 243 KSP preconditioned resid norm 3.878468605244e-03 true resid norm 9.538994838179e-06 ||r(i)||/||b|| 1.450512003856e-03 244 KSP Residual norm 1.006470041466e-03 244 KSP preconditioned resid norm 1.006470041466e-03 true resid norm 2.630164744018e-06 ||r(i)||/||b|| 3.999462834436e-04 245 KSP Residual norm 1.318278819855e-03 245 KSP preconditioned resid norm 1.318278819855e-03 true resid norm 4.012629952397e-06 ||r(i)||/||b|| 6.101657472012e-04 246 KSP Residual norm 1.025266732268e-03 246 KSP preconditioned resid norm 1.025266732268e-03 true resid norm 2.539814594543e-06 ||r(i)||/||b|| 3.862075218041e-04 247 KSP Residual norm 8.239890273661e-04 247 KSP preconditioned resid norm 8.239890273661e-04 true resid norm 3.052252843148e-06 ||r(i)||/||b|| 4.641295506389e-04 248 KSP Residual norm 7.725105959877e-04 248 KSP preconditioned resid norm 7.725105959877e-04 true resid norm 1.985956547924e-06 ||r(i)||/||b|| 3.019871444287e-04 249 KSP Residual norm 7.574072520328e-04 249 KSP preconditioned resid norm 7.574072520328e-04 true resid norm 1.899524424100e-06 ||r(i)||/||b|| 2.888441628826e-04 250 KSP Residual norm 7.416031935167e-04 250 KSP preconditioned resid norm 7.416031935167e-04 true resid norm 1.904708263813e-06 ||r(i)||/||b|| 2.896324243145e-04 251 KSP Residual norm 7.377911096787e-04 251 KSP preconditioned resid norm 7.377911096787e-04 true resid norm 1.835116039699e-06 ||r(i)||/||b|| 2.790501398952e-04 252 KSP Residual norm 7.202907178956e-04 252 KSP preconditioned resid norm 7.202907178956e-04 true resid norm 1.780562119895e-06 ||r(i)||/||b|| 2.707545996547e-04 253 KSP Residual norm 8.394896961152e-04 253 KSP preconditioned resid norm 8.394896961152e-04 true resid norm 2.256975188767e-06 ||r(i)||/||b|| 3.431985926452e-04 254 KSP Residual norm 6.433289364997e-04 254 KSP preconditioned resid norm 6.433289364997e-04 true resid norm 1.631114620709e-06 ||r(i)||/||b|| 2.480294178936e-04 255 KSP Residual norm 5.843672322804e-04 255 KSP preconditioned resid norm 5.843672322804e-04 true resid norm 1.715009320542e-06 ||r(i)||/||b|| 2.607865554362e-04 256 KSP Residual norm 5.769222686234e-04 256 KSP preconditioned resid norm 5.769222686234e-04 true resid norm 1.690631052069e-06 ||r(i)||/||b|| 2.570795641176e-04 257 KSP Residual norm 5.398934551490e-04 257 KSP preconditioned resid norm 5.398934551490e-04 true resid norm 1.379874733785e-06 ||r(i)||/||b|| 2.098255528100e-04 258 KSP Residual norm 5.557798491806e-04 258 KSP preconditioned resid norm 5.557798491806e-04 true resid norm 1.634221862103e-06 ||r(i)||/||b|| 2.485019090751e-04 259 KSP Residual norm 5.420364606578e-04 259 KSP preconditioned resid norm 5.420364606578e-04 true resid norm 1.461478523529e-06 ||r(i)||/||b|| 2.222343315747e-04 260 KSP Residual norm 5.313021747502e-04 260 KSP preconditioned resid norm 5.313021747502e-04 true resid norm 1.345861639160e-06 ||r(i)||/||b|| 2.046534772529e-04 261 KSP Residual norm 5.300120886355e-04 261 KSP preconditioned resid norm 5.300120886355e-04 true resid norm 1.365448063251e-06 ||r(i)||/||b|| 2.076318144614e-04 262 KSP Residual norm 5.327770811846e-04 262 KSP preconditioned resid norm 5.327770811846e-04 true resid norm 1.332230585303e-06 ||r(i)||/||b|| 2.025807214145e-04 263 KSP Residual norm 5.336599225908e-04 263 KSP preconditioned resid norm 5.336599225908e-04 true resid norm 1.355936135446e-06 ||r(i)||/||b|| 2.061854183058e-04 264 KSP Residual norm 5.355215363839e-04 264 KSP preconditioned resid norm 5.355215363839e-04 true resid norm 1.330340515039e-06 ||r(i)||/||b|| 2.022933148635e-04 265 KSP Residual norm 5.428455910138e-04 265 KSP preconditioned resid norm 5.428455910138e-04 true resid norm 1.404246142210e-06 ||r(i)||/||b|| 2.135315009807e-04 266 KSP Residual norm 5.612005666228e-04 266 KSP preconditioned resid norm 5.612005666228e-04 true resid norm 1.436045514368e-06 ||r(i)||/||b|| 2.183669550106e-04 267 KSP Residual norm 5.578420720622e-04 267 KSP preconditioned resid norm 5.578420720622e-04 true resid norm 1.430268089908e-06 ||r(i)||/||b|| 2.174884323075e-04 268 KSP Residual norm 5.602687994241e-04 268 KSP preconditioned resid norm 5.602687994241e-04 true resid norm 1.393656233491e-06 ||r(i)||/||b|| 2.119211856407e-04 269 KSP Residual norm 5.561141727370e-04 269 KSP preconditioned resid norm 5.561141727370e-04 true resid norm 1.389866310789e-06 ||r(i)||/||b|| 2.113448850487e-04 270 KSP Residual norm 5.703325128907e-04 270 KSP preconditioned resid norm 5.703325128907e-04 true resid norm 1.508707692704e-06 ||r(i)||/||b|| 2.294160606754e-04 271 KSP Residual norm 5.831128974940e-04 271 KSP preconditioned resid norm 5.831128974940e-04 true resid norm 2.247862938089e-06 ||r(i)||/||b|| 3.418129719153e-04 272 KSP Residual norm 5.578052888760e-04 272 KSP preconditioned resid norm 5.578052888760e-04 true resid norm 1.474365217798e-06 ||r(i)||/||b|| 2.241938991229e-04 273 KSP Residual norm 5.735144663521e-04 273 KSP preconditioned resid norm 5.735144663521e-04 true resid norm 1.626062556386e-06 ||r(i)||/||b|| 2.472611943996e-04 274 KSP Residual norm 5.939683728948e-04 274 KSP preconditioned resid norm 5.939683728948e-04 true resid norm 1.491326544822e-06 ||r(i)||/||b|| 2.267730606453e-04 275 KSP Residual norm 6.847568949980e-04 275 KSP preconditioned resid norm 6.847568949980e-04 true resid norm 1.702016830551e-06 ||r(i)||/||b|| 2.588109004524e-04 276 KSP Residual norm 6.764683670568e-04 276 KSP preconditioned resid norm 6.764683670568e-04 true resid norm 1.680400141222e-06 ||r(i)||/||b|| 2.555238384623e-04 277 KSP Residual norm 6.389674111349e-04 277 KSP preconditioned resid norm 6.389674111349e-04 true resid norm 1.577941565938e-06 ||r(i)||/||b|| 2.399438537922e-04 278 KSP Residual norm 2.773202475295e-04 278 KSP preconditioned resid norm 2.773202475295e-04 true resid norm 6.894432844176e-07 ||r(i)||/||b|| 1.048376455791e-04 279 KSP Residual norm 2.148994428870e-04 279 KSP preconditioned resid norm 2.148994428870e-04 true resid norm 5.335839647283e-07 ||r(i)||/||b|| 8.113747402462e-05 280 KSP Residual norm 2.528602731064e-04 280 KSP preconditioned resid norm 2.528602731064e-04 true resid norm 6.249333257144e-07 ||r(i)||/||b|| 9.502817707067e-05 281 KSP Residual norm 1.232370379443e-04 281 KSP preconditioned resid norm 1.232370379443e-04 true resid norm 3.044944566605e-07 ||r(i)||/||b|| 4.630182445702e-05 282 KSP Residual norm 1.413398433340e-04 282 KSP preconditioned resid norm 1.413398433340e-04 true resid norm 3.491514577577e-07 ||r(i)||/||b|| 5.309242632300e-05 283 KSP Residual norm 9.858967955326e-05 283 KSP preconditioned resid norm 9.858967955326e-05 true resid norm 2.585549572093e-07 ||r(i)||/||b|| 3.931620421762e-05 284 KSP Residual norm 6.153061004992e-05 284 KSP preconditioned resid norm 6.153061004992e-05 true resid norm 1.532327761243e-07 ||r(i)||/||b|| 2.330077591225e-05 285 KSP Residual norm 8.010218213372e-05 285 KSP preconditioned resid norm 8.010218213372e-05 true resid norm 1.980252596864e-07 ||r(i)||/||b|| 3.011197941866e-05 286 KSP Residual norm 7.745411900644e-05 286 KSP preconditioned resid norm 7.745411900644e-05 true resid norm 1.966883638978e-07 ||r(i)||/||b|| 2.990868930034e-05 287 KSP Residual norm 7.337483421192e-05 287 KSP preconditioned resid norm 7.337483421192e-05 true resid norm 1.825378726900e-07 ||r(i)||/||b|| 2.775694714034e-05 288 KSP Residual norm 8.298054858404e-05 288 KSP preconditioned resid norm 8.298054858404e-05 true resid norm 2.073501002277e-07 ||r(i)||/||b|| 3.152992684065e-05 289 KSP Residual norm 1.499466977520e-03 289 KSP preconditioned resid norm 1.499466977520e-03 true resid norm 8.925790709540e-06 ||r(i)||/||b|| 1.357267383800e-03 290 KSP Residual norm 4.695309564937e-04 290 KSP preconditioned resid norm 4.695309564937e-04 true resid norm 1.165519784197e-06 ||r(i)||/||b|| 1.772304594341e-04 291 KSP Residual norm 4.100795617257e-03 291 KSP preconditioned resid norm 4.100795617257e-03 true resid norm 1.014254057847e-05 ||r(i)||/||b|| 1.542287956776e-03 292 KSP Residual norm 9.465543603790e-05 292 KSP preconditioned resid norm 9.465543603790e-05 true resid norm 2.472672993454e-07 ||r(i)||/||b|| 3.759978823200e-05 293 KSP Residual norm 1.141311980908e-04 293 KSP preconditioned resid norm 1.141311980908e-04 true resid norm 2.815266092808e-07 ||r(i)||/||b|| 4.280930361051e-05 294 KSP Residual norm 9.618463858315e-06 294 KSP preconditioned resid norm 9.618463858315e-06 true resid norm 2.458825950866e-08 ||r(i)||/||b|| 3.738922829532e-06 2 SNES Function norm 2.458829523543e-08 Number of KSP iteration is 294 SNES solve takes time 76.5517914635294119 Number of Newton iterations = 2 the SNES Converged Reason is 3 the memory used finally in SolveNE_option is 1046126592.00000000 in 0 th processor solve_NE for flow equations took time 76.5565609905881956 From jedbrown at mcs.anl.gov Sat Jun 30 04:53:05 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 30 Jun 2012 01:53:05 -0800 Subject: [petsc-users] Any suggestions for improving the convergence rate or performance? In-Reply-To: References: Message-ID: Try using a multilevel method for the Poisson solve. If you use DM, you can just do -pc_type mg -pc_mg_levels 4 (or whatever) to do geometric multigrid. Otherwise, start with algebraic multigrid, -pc_type gamg, or configure with ML or Hypre and -pc_type ml or -pc_type hypre. The iteration count should be much lower, though each iteration will be somewhat more expensive. On Sat, Jun 30, 2012 at 12:16 AM, Bao Kai wrote: > Hi, all, > > I am currently trying to solve a solute transport problem with PETSC > and finite difference method with structured grids. Two equations are > involved, one is a transport equation with advection and diffusion > terms, another one is the flow equation with gravity, which is > actually a poisson equation. Both the equations use the same solver. > > Since we eventually want to handle non-linear problems, we us SNES > here to solve the equations. I use the BICGSTAB ksp linear solver and > block Jacobi preconditioner. All the tolerances are the default ones > from PETSC. The range of the solution values for transport equation > is about 0-1100, and the range of the solution values of the flow > equations should be between -5000-5000. > > The problem is that the solution of flow equations takes about ten > times of the time than the transport equations. I am wondering if I > should set different tolerances for the two equations. > > Attached please find the convergence processes of the two equations. > Generally it takes about ten times of iterations to solve the flow > equations than transport equations. I only do applications and do not > have good knowledge in the solution of linear system, so I am here to > find some suggestions on this issue. > > The size of the mesh is 500X500X500 (125 million unknowns ) and 512 > nodes (2048 processors ) on BlueGene/P machine are used. > > Any suggestions will be much appreciated. > > Thank you . > > Kai > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jun 30 06:30:03 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 30 Jun 2012 05:30:03 -0600 Subject: [petsc-users] about parallel preconditioned matrix-free gmres In-Reply-To: References: Message-ID: On Fri, Jun 29, 2012 at 7:25 PM, RenZhengYong wrote: > Dear Petscs, > > Use the uniprocessor complex-value based version petsc, I recently > successfully make a FETI_DP domain > decomposition approach working for 3D electromagnetic induction (earth) > problem. The number of iteration of > the interface problem seems to be scalable with regard to the number of > sub-domains. > > To do this, I had two subroutines for petsc > > (1) int mat_vec_product_interface_problem(Mat A, Vec X, Vec Y) for > matrix-free GMRES solver > (2) int preconditioner_mat_vec(PC pc,Vec X,Vec Y) for shell > preconditioner. > > Now, I want to solve the interface problem by paralleled GMRES solver so > that I can solve real large-scale problems. Could you please tell me the > easiest way to accomplish it. Which specific data structures of petsc > should be used. I have been using Petsc for 3.5 years, I really want to > have a try the real MPI-based Petsc. > 1) The solver logic should be parallel already since it only uses calls to Vec or Mat functions. The problems will be in building data structures. 2) It looks like your two items above are the things to be parallelized 3) Decide how to partition the problem 4) Use VecScatter() to communicate data along the interface of your partitions I don't think we can give better advice than that without more specific questions. Note that there is a current effort to put BDDCinto PETSc. You can see it in petsc-dev, as PCBDDC. Thanks, Matt > Thanks in advance. > Have a nice weekeed > Zhengyong > > > -- > Zhengyong Ren > AUG Group, Institute of Geophysics > Department of Geosciences, ETH Zurich > NO H 47 Sonneggstrasse 5 > CH-8092, Z?rich, Switzerland > Tel: +41 44 633 37561 > e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch > Gmail: renzhengyong at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Sat Jun 30 06:39:16 2012 From: renzhengyong at gmail.com (RenZhengYong) Date: Sat, 30 Jun 2012 13:39:16 +0200 Subject: [petsc-users] about parallel preconditioned matrix-free gmres In-Reply-To: References: Message-ID: Hi, Matt, Thanks a lot for your suggestions. I will try it today and once I had specific questions, I mightbe needs your helps. Best wishes Zhengyong On Sat, Jun 30, 2012 at 1:30 PM, Matthew Knepley wrote: > On Fri, Jun 29, 2012 at 7:25 PM, RenZhengYong wrote: > >> Dear Petscs, >> >> Use the uniprocessor complex-value based version petsc, I recently >> successfully make a FETI_DP domain >> decomposition approach working for 3D electromagnetic induction (earth) >> problem. The number of iteration of >> the interface problem seems to be scalable with regard to the number of >> sub-domains. >> >> To do this, I had two subroutines for petsc >> >> (1) int mat_vec_product_interface_problem(Mat A, Vec X, Vec Y) for >> matrix-free GMRES solver >> (2) int preconditioner_mat_vec(PC pc,Vec X,Vec Y) for shell >> preconditioner. >> >> Now, I want to solve the interface problem by paralleled GMRES solver so >> that I can solve real large-scale problems. Could you please tell me the >> easiest way to accomplish it. Which specific data structures of petsc >> should be used. I have been using Petsc for 3.5 years, I really want to >> have a try the real MPI-based Petsc. >> > > 1) The solver logic should be parallel already since it only uses calls to > Vec or Mat functions. The problems will be > in building data structures. > > 2) It looks like your two items above are the things to be parallelized > > 3) Decide how to partition the problem > > 4) Use VecScatter() to communicate data along the interface of your > partitions > > I don't think we can give better advice than that without more specific > questions. Note that there is > a current effort to put BDDCinto PETSc. You can see it in petsc-dev, as > PCBDDC. > > Thanks, > > Matt > > >> Thanks in advance. >> Have a nice weekeed >> Zhengyong >> >> >> -- >> Zhengyong Ren >> AUG Group, Institute of Geophysics >> Department of Geosciences, ETH Zurich >> NO H 47 Sonneggstrasse 5 >> CH-8092, Z?rich, Switzerland >> Tel: +41 44 633 37561 >> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch >> Gmail: renzhengyong at gmail.com >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Zhengyong Ren AUG Group, Institute of Geophysics Department of Geosciences, ETH Zurich NO H 47 Sonneggstrasse 5 CH-8092, Z?rich, Switzerland Tel: +41 44 633 37561 e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch Gmail: renzhengyong at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: