From u.tabak at tudelft.nl Mon Aug 3 05:07:39 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 12:07:39 +0200 Subject: tolerance for diagonal entries Message-ID: <4A76B6EB.2060907@tudelft.nl> Dear all, I am reading some matrices into petsc, but some of the diagonal entries are really small, on the order of e-8. Petsc complains about missing diagonal entries, I guess it compares them with some predefined tolerance value, is there a way to change/set these tolerances? Error is "Matrix is missing diagonal entry 1847!" after read from a matrix market formatted text file. Best regards, Umut From u.tabak at tudelft.nl Mon Aug 3 06:44:32 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 13:44:32 +0200 Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A76B6EB.2060907@tudelft.nl> References: <4A76B6EB.2060907@tudelft.nl> Message-ID: <4A76CDA0.3060704@tudelft.nl> Umut Tabak wrote: > Dear all, > > I am reading some matrices into petsc, but some of the diagonal > entries are really small, on the order of e-8. Petsc complains about > missing diagonal entries, I guess it compares them with some > predefined tolerance value, is there a way to change/set these > tolerances? > > Error is "Matrix is missing diagonal entry 1847!" after read from a > matrix market formatted text file. > > Best regards, > Umut > > sorry that I forgot to explain my problem completely, I get this error when I try to use Slepc's EPSSolve function, so that the question is also related to Slepc I guess. I mean the tolerancing issues for the matrices. From xy2102 at columbia.edu Mon Aug 3 08:04:56 2009 From: xy2102 at columbia.edu ((Rebecca) Xuefei YUAN) Date: Mon, 03 Aug 2009 09:04:56 -0400 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? Message-ID: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> Thanks, -- (Rebecca) Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From hzhang at mcs.anl.gov Mon Aug 3 09:13:31 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 3 Aug 2009 09:13:31 -0500 (CDT) Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A76CDA0.3060704@tudelft.nl> References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> Message-ID: Umut, The error msg "Matrix is missing diagonal entry 1847!" has nothing to do with the tolerance. It indicates that you did not input any value to the 1847-th diagonal entry. Many routines required diagonal entries been allocated and set. If that entry is zero, you should set zero value to it. Hong On Mon, 3 Aug 2009, Umut Tabak wrote: > Umut Tabak wrote: >> Dear all, >> >> I am reading some matrices into petsc, but some of the diagonal entries are >> really small, on the order of e-8. Petsc complains about missing diagonal >> entries, I guess it compares them with some predefined tolerance value, is >> there a way to change/set these tolerances? >> >> Error is "Matrix is missing diagonal entry 1847!" after read from a matrix >> market formatted text file. >> >> Best regards, >> Umut >> >> > sorry that I forgot to explain my problem completely, I get this error when I > try to use Slepc's EPSSolve function, so that the question is also related to > Slepc I guess. I mean the tolerancing issues for the matrices. > > From u.tabak at tudelft.nl Mon Aug 3 09:38:16 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 16:38:16 +0200 Subject: tolerance for diagonal entries + Slepc In-Reply-To: References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> Message-ID: <4A76F658.7020305@tudelft.nl> Hong Zhang wrote: > > Umut, > The error msg > "Matrix is missing diagonal entry 1847!" > has nothing to do with the tolerance. > It indicates that you did not input any value > to the 1847-th diagonal entry. > Many routines required diagonal entries been > allocated and set. > If that entry is zero, you should set zero value to it. > > Hong > Hi Hong, Thanks for the quick reply, that last part of the matrix in matrix market format, I read this with one of the examples that you can read in the matrix market real format: ... 1848 925 1.207005667e-06 1848 1032 1.545494254e-08 1848 1771 1.207005667e-06 1848 1777 4.828022669e-06 1848 1782 -3.078839811e-08 1848 1842 1.545494254e-08 1848 1848 6.181977018e-08 // this is source of the issue ... So actually very close to zero. I have this numerical problem on most of my matrices that I would like to interface with Petsc. The point which made me think this way is that Matlab which uses Arpack to solve eigenvalue problems did not experience any problems to solve the system. In Petsc+Slepc, I ran into this problem(a generalized eigenvalue problem,BTW). Best regards, Umut From mossaiby at yahoo.com Mon Aug 3 09:43:30 2009 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Mon, 3 Aug 2009 07:43:30 -0700 (PDT) Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A76F658.7020305@tudelft.nl> Message-ID: <54519.6793.qm@web52201.mail.re2.yahoo.com> Matrix Market indices are 1-based. Please keep this in mind when searching. --- On Mon, 8/3/09, Umut Tabak wrote: > From: Umut Tabak > Subject: Re: tolerance for diagonal entries + Slepc > To: "PETSc users list" > Date: Monday, August 3, 2009, 7:08 PM > Hong Zhang wrote: > > > > Umut, > > The error msg > > "Matrix is missing diagonal entry 1847!" > > has nothing to do with the tolerance. > > It indicates that you did not input any value > > to the 1847-th diagonal entry. > > Many routines required diagonal entries been > > allocated and set. > > If that entry is zero, you should set zero value to > it. > > > > Hong > > > Hi Hong, > > Thanks for the quick reply, that last part of the matrix in > matrix market format, I read this with one of the examples > that you can read in the matrix market real format: > ... > 1848 925 1.207005667e-06 > 1848 1032 1.545494254e-08 > 1848 1771 1.207005667e-06 > 1848 1777 4.828022669e-06 > 1848 1782 -3.078839811e-08 > 1848 1842 1.545494254e-08 > 1848 1848 6.181977018e-08 // this is source of the issue > ... > So actually very close to zero. I have this numerical > problem on most of my matrices that I would like to > interface with Petsc. The point which made me think this way > is that Matlab which uses Arpack to solve eigenvalue > problems did not experience any problems to solve the > system. In Petsc+Slepc, I ran into this problem(a > generalized eigenvalue problem,BTW). > > Best regards, > Umut > > > From hzhang at mcs.anl.gov Mon Aug 3 09:55:31 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 3 Aug 2009 09:55:31 -0500 (CDT) Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A76F658.7020305@tudelft.nl> References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> Message-ID: > Hi Hong, > > Thanks for the quick reply, that last part of the matrix in matrix market > format, I read this with one of the examples that you can read in the matrix > market real format: > ... > 1848 925 1.207005667e-06 > 1848 1032 1.545494254e-08 > 1848 1771 1.207005667e-06 > 1848 1777 4.828022669e-06 > 1848 1782 -3.078839811e-08 > 1848 1842 1.545494254e-08 > 1848 1848 6.181977018e-08 // this is source of the issue > ... > So actually very close to zero. I have this numerical problem on most of my > matrices that I would like to interface with Petsc. The point which made me > think this way is that Matlab which uses Arpack to solve eigenvalue problems > did not experience any problems to solve the system. In Petsc+Slepc, I ran > into this problem(a generalized eigenvalue problem,BTW). Which code do you use for reading the matrix in matrix market into petsc? I guess that code needs to be modified. Hong > > Best regards, > Umut > > > From u.tabak at tudelft.nl Mon Aug 3 10:20:11 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 17:20:11 +0200 Subject: tolerance for diagonal entries + Slepc In-Reply-To: References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> Message-ID: <4A77002B.6080109@tudelft.nl> Hong Zhang wrote: > > > Which code do you use for reading the matrix in matrix market > into petsc? I guess that code needs to be modified. > > Hong > I used ex72.c in src/mat/examples/tests where there is a formatted read operation with fscanf. The format is specified with le for doubles, if I am not mistaken... From hzhang at mcs.anl.gov Mon Aug 3 10:34:32 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 3 Aug 2009 10:34:32 -0500 (CDT) Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A77002B.6080109@tudelft.nl> References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> <4A77002B.6080109@tudelft.nl> Message-ID: May I have this matrix to test ex72.c? Hong On Mon, 3 Aug 2009, Umut Tabak wrote: > Hong Zhang wrote: >> >> >> Which code do you use for reading the matrix in matrix market >> into petsc? I guess that code needs to be modified. >> >> Hong >> > I used ex72.c in src/mat/examples/tests where there is a formatted read > operation with fscanf. The format is specified with le for doubles, if I am > not mistaken... > From u.tabak at tudelft.nl Mon Aug 3 10:45:26 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 17:45:26 +0200 Subject: tolerance for diagonal entries + Slepc In-Reply-To: References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> <4A77002B.6080109@tudelft.nl> Message-ID: <4A770616.1020108@tudelft.nl> Hong Zhang wrote: > > May I have this matrix to test ex72.c? > Hong Thanks, in the attach you find the matrix... -------------- next part -------------- A non-text attachment was scrubbed... Name: Mshell.mm Type: text/x-troff-mm Size: 433279 bytes Desc: not available URL: From hzhang at mcs.anl.gov Mon Aug 3 11:28:38 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 3 Aug 2009 11:28:38 -0500 (CDT) Subject: tolerance for diagonal entries + Slepc In-Reply-To: <4A770616.1020108@tudelft.nl> References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> <4A77002B.6080109@tudelft.nl> <4A770616.1020108@tudelft.nl> Message-ID: Umut, The data file Mshell.mm has incorrect nnz = 14865. 'wc -l Mshell.mm' displays 17571 Mshell.mm i.e., nnz = 17571 - 2 ( num of lines - 2 header lines) I add if (i == nnz-1) printf("i %d, row %d, col %d\n",i,row,col); in line 58 of ex72.c, and get ./ex72 -fin Mshell.mm -fout mat.out %% Matrix Market Real Matrix m = 1848, n = 1848, nnz = 14865 i 14864, row 1553, col 1487 Reading matrix completes. As you see, you only read matrix upto row 1553, missing many entries. I notice that the matrix does not have any entry in row 1846: 1845 1839 0.0006074348926 1845 1845 0.00242973957 1847 925 -1.207013266e-06 1847 1027 -4.828053065e-06 This is a singular matrix. Petsc solver will complain about it. Hong On Mon, 3 Aug 2009, Umut Tabak wrote: > Hong Zhang wrote: >> >> May I have this matrix to test ex72.c? >> Hong > Thanks, in the attach you find the matrix... > From u.tabak at tudelft.nl Mon Aug 3 12:04:27 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Mon, 03 Aug 2009 19:04:27 +0200 Subject: tolerance for diagonal entries + Slepc In-Reply-To: References: <4A76B6EB.2060907@tudelft.nl> <4A76CDA0.3060704@tudelft.nl> <4A76F658.7020305@tudelft.nl> <4A77002B.6080109@tudelft.nl> <4A770616.1020108@tudelft.nl> Message-ID: <4A77189B.3050600@tudelft.nl> Hong Zhang wrote: > > Umut, > > The data file Mshell.mm has incorrect nnz = 14865. > 'wc -l Mshell.mm' displays > > 17571 Mshell.mm > i.e., nnz = 17571 - 2 ( num of lines - 2 header lines) > > I add > if (i == nnz-1) printf("i %d, row %d, col %d\n",i,row,col); > in line 58 of ex72.c, and get > > ./ex72 -fin Mshell.mm -fout mat.out > %% Matrix Market Real Matrix > m = 1848, n = 1848, nnz = 14865 > i 14864, row 1553, col 1487 > Reading matrix completes. > > As you see, you only read matrix upto row 1553, missing > many entries. > Thanks for the pointers, actually, the matrices are exported from a program on a tolerance selection, I guess there is a problem on the selection of that tolerance value or the implementation, I did not think that the matrix export part could have problems, honestly. So thanks for the pointers and debugging help. From bsmith at mcs.anl.gov Mon Aug 3 12:52:19 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 3 Aug 2009 12:52:19 -0500 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? In-Reply-To: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> References: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> Message-ID: <4ED6AE3A-BEDC-4CC1-8650-D0F0A0CE3678@mcs.anl.gov> Thanks for reporting this. The documentation for this Matlab function is wrong, it doesn't actually return anything. I have fixed the docs and it will be fixed in the next patch. Barry On Aug 3, 2009, at 8:04 AM, (Rebecca) Xuefei YUAN wrote: > > Thanks, > > -- > (Rebecca) Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > From Harun.BAYRAKTAR at 3ds.com Mon Aug 3 13:09:34 2009 From: Harun.BAYRAKTAR at 3ds.com (BAYRAKTAR Harun) Date: Mon, 3 Aug 2009 14:09:34 -0400 Subject: Smoother settings for AMG In-Reply-To: References: Message-ID: Hi Barry, I reproduced the issue using the binary I sent you earlier and using ex10. I sent an e-mail to petsc-maint with an attachment that has all options used and the output with convergence info, ksp view etc. Thanks, Harun -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith Sent: Friday, July 31, 2009 2:25 PM To: PETSc users list Subject: Re: Smoother settings for AMG On Jul 31, 2009, at 1:15 PM, BAYRAKTAR Harun wrote: > Barry, > > Thanks a lot for looking in to this. One thing I want to clarify is > that the 43 (should have been 46 sorry for the typo) iterations on 1 > cpu seems like a real convergence to me. I do look at the > unpreconditioned residual norm to determine convergence. For this I > use: > > ierr = KSPSetNormType(m_solver, KSP_NORM_UNPRECONDITIONED); > CHKERRQ(ierr); > > Then I check convergence through KSPSetConvergenceTest. As an > experiment I commented out the line above where I tell KSP to use > the unpreconditioned norm and while the ||r|| values changed > (naturally), it still converged in slightly more number of > iterations (56). > > I am familiar with the preconditioned norm going down 6 orders while > the true relative norm is 0.1 or so (i.e., problem not solved at > all). This usually happens to me in structural mechanics problems > with ill conditioned systems and I use a KSP method that does not > allow for the unpreconditioned residual to be monitored. However, > this does not seem to be one of those cases though, maybe I am > missing something. Ok. I didn't see what you report (I saw it just iterating away for a long time with the unpreconditioned norm) but then you never sent the command line options for the solver you used so I may have run it differently. > > Out of curiosity did you use ksp/ksp/examples/tutorials/ex10.c to > solve this? Yes. > > Thanks again, > Harun > > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov > ] On Behalf Of Barry Smith > Sent: Thursday, July 30, 2009 7:45 PM > To: PETSc users list > Subject: Re: Smoother settings for AMG > > > Harun, > > I have played around with this matrix. It is a nasty matrix; I > think it is really beyond the normal capacity of ML (and hypre's > boomerAMG). > > Even the "convergence" you were getting below is BOGUS. If you run > with -ksp_norm_type unpreconditioned or -ksp_monitor_true_residual > you'll see that the "true" residual norm is actually creeping to zero > and at the converged 43 iterations below the true residual norm has > decreased by like less than 1/10. (The preconditioned residual norm > has decreased by 1.e 5 so the iteration stops and you think it has > converged. In really hard problems preconditioners sometimes scales > things in a funky way so a large decrease in preconditioned residual > norm does not mean a large decrease in true residual norm). In other > words the "answer" you got out of the runs below is garbage. > > I suggest, > 1) check carefully that the matrix being created actually matches the > model's equations, if they seem right then > 2) see if you can change the model so it does not generate such > hopeless matrices. If you MUST solve this nasty matrix > 3) bite the bullet and use a parallel direct solver from PETSc. Try > both MUMPS and SuperLU_dist > > Good luck, > > Barry > > > > > On Jul 29, 2009, at 3:54 PM, BAYRAKTAR Harun wrote: > >> Hi, >> >> I am trying to solve a system of equations and I am having difficulty >> picking the right smoothers for AMG (using ML as pc_type) in PETSc >> for >> parallel execution. First here is what happens in terms of CG >> (ksp_type) >> iteration counts (both columns use block jacobi): >> >> cpus | AMG w/ ICC(0) x1 | AMG w/ SOR x4 >> ------------------------------------------------------ >> 1 | 43 | 243 >> 4 | 699 | 379 >> >> x1 or x4 means 1 or 4 iterations of smoother application at each AMG >> level (all details from ksp view for the 4 cpu run are below). The >> main >> observation is that on 1 cpu, AMG w/ ICC(0) is a clear winner but >> falls >> apart in parallel. SOR on the other hand experiences a 1.5X increase >> in >> iteration count which is totally expected from the quality of >> coarsening >> ML delivers in parallel. >> >> I basically would like to find a way (if possible) to have the >> number of >> iterations in parallel stay with 1-2X of 1 cpu iteration count for >> the >> AMG w/ ICC case. Is there a way to achieve this? >> >> Thanks, >> Harun >> >> %%%%%%%%%%%%%%%%%%%%%%%%% >> AMG w/ ICC(0) x1 ksp_view >> %%%%%%%%%%%%%%%%%%%%%%%%% >> KSP Object: >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object: >> type: ml >> MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1, >> post-smooths=1 >> Coarse gride solver -- level 0 ------------------------------- >> KSP Object:(mg_coarse_) >> type: preonly >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_coarse_) >> type: redundant >> Redundant preconditioner: First (color=0) of 4 PCs follows >> KSP Object:(mg_coarse_redundant_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_coarse_redundant_) >> type: lu >> LU: out-of-place factorization >> matrix ordering: nd >> LU: tolerance for zero pivot 1e-12 >> LU: factor fill ratio needed 2.17227 >> Factored matrix follows >> Matrix Object: >> type=seqaij, rows=283, cols=283 >> total: nonzeros=21651, allocated nonzeros=21651 >> using I-node routines: found 186 nodes, limit used is >> 5 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=283, cols=283 >> total: nonzeros=9967, allocated nonzeros=14150 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=283, cols=283 >> total: nonzeros=9967, allocated nonzeros=9967 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object:(mg_levels_1_) >> type: richardson >> Richardson: damping factor=0.9 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_) >> type: bjacobi >> block Jacobi: number of blocks = 4 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object:(mg_levels_1_sub_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_sub_) >> type: icc >> ICC: 0 levels of fill >> ICC: factor fill ratio allocated 1 >> ICC: using Manteuffel shift >> ICC: factor fill ratio needed 0.514899 >> Factored matrix follows >> Matrix Object: >> type=seqsbaij, rows=2813, cols=2813 >> total: nonzeros=48609, allocated nonzeros=48609 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=2813, cols=2813 >> total: nonzeros=94405, allocated nonzeros=94405 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=10654, cols=10654 >> total: nonzeros=376634, allocated nonzeros=376634 >> not using I-node (on process 0) routines >> Up solver (post-smoother) on level 1 ------------------------------- >> KSP Object:(mg_levels_1_) >> type: richardson >> Richardson: damping factor=0.9 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_) >> type: bjacobi >> block Jacobi: number of blocks = 4 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object:(mg_levels_1_sub_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_sub_) >> type: icc >> ICC: 0 levels of fill >> ICC: factor fill ratio allocated 1 >> ICC: using Manteuffel shift >> ICC: factor fill ratio needed 0.514899 >> Factored matrix follows >> Matrix Object: >> type=seqsbaij, rows=2813, cols=2813 >> total: nonzeros=48609, allocated nonzeros=48609 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=2813, cols=2813 >> total: nonzeros=94405, allocated nonzeros=94405 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=10654, cols=10654 >> total: nonzeros=376634, allocated nonzeros=376634 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 2 ------------------------------- >> KSP Object:(mg_levels_2_) >> type: richardson >> Richardson: damping factor=0.9 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_) >> type: bjacobi >> block Jacobi: number of blocks = 4 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object:(mg_levels_2_sub_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_sub_) >> type: icc >> ICC: 0 levels of fill >> ICC: factor fill ratio allocated 1 >> ICC: using Manteuffel shift >> ICC: factor fill ratio needed 0.519045 >> Factored matrix follows >> Matrix Object: >> type=seqsbaij, rows=101164, cols=101164 >> total: nonzeros=1378558, allocated nonzeros=1378558 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=101164, cols=101164 >> total: nonzeros=2655952, allocated nonzeros=5159364 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> Up solver (post-smoother) on level 2 ------------------------------- >> KSP Object:(mg_levels_2_) >> type: richardson >> Richardson: damping factor=0.9 >> maximum iterations=1 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_) >> type: bjacobi >> block Jacobi: number of blocks = 4 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object:(mg_levels_2_sub_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_sub_) >> type: icc >> ICC: 0 levels of fill >> ICC: factor fill ratio allocated 1 >> ICC: using Manteuffel shift >> ICC: factor fill ratio needed 0.519045 >> Factored matrix follows >> Matrix Object: >> type=seqsbaij, rows=101164, cols=101164 >> total: nonzeros=1378558, allocated nonzeros=1378558 >> block size is 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=101164, cols=101164 >> total: nonzeros=2655952, allocated nonzeros=5159364 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> >> %%%%%%%%%%%%%%%%%%%%%% >> AMG w/ SOR x4 ksp_view >> %%%%%%%%%%%%%%%%%%%%%% >> >> KSP Object: >> type: cg >> maximum iterations=10000 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object: >> type: ml >> MG: type is MULTIPLICATIVE, levels=3 cycles=v, pre-smooths=1, >> post-smooths=1 >> Coarse gride solver -- level 0 ------------------------------- >> KSP Object:(mg_coarse_) >> type: preonly >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_coarse_) >> type: redundant >> Redundant preconditioner: First (color=0) of 4 PCs follows >> KSP Object:(mg_coarse_redundant_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_coarse_redundant_) >> type: lu >> LU: out-of-place factorization >> matrix ordering: nd >> LU: tolerance for zero pivot 1e-12 >> LU: factor fill ratio needed 2.17227 >> Factored matrix follows >> Matrix Object: >> type=seqaij, rows=283, cols=283 >> total: nonzeros=21651, allocated nonzeros=21651 >> using I-node routines: found 186 nodes, limit used is >> 5 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=283, cols=283 >> total: nonzeros=9967, allocated nonzeros=14150 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=283, cols=283 >> total: nonzeros=9967, allocated nonzeros=9967 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object:(mg_levels_1_) >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=4, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_) >> type: sor >> SOR: type = local_symmetric, iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=10654, cols=10654 >> total: nonzeros=376634, allocated nonzeros=376634 >> not using I-node (on process 0) routines >> Up solver (post-smoother) on level 1 ------------------------------- >> KSP Object:(mg_levels_1_) >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=4 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_1_) >> type: sor >> SOR: type = local_symmetric, iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=10654, cols=10654 >> total: nonzeros=376634, allocated nonzeros=376634 >> not using I-node (on process 0) routines >> Down solver (pre-smoother) on level 2 ------------------------------- >> KSP Object:(mg_levels_2_) >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=4, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_) >> type: sor >> SOR: type = local_symmetric, iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> Up solver (post-smoother) on level 2 ------------------------------- >> KSP Object:(mg_levels_2_) >> type: richardson >> Richardson: damping factor=1 >> maximum iterations=4 >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_2_) >> type: sor >> SOR: type = local_symmetric, iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=mpiaij, rows=411866, cols=411866 >> total: nonzeros=10941434, allocated nonzeros=42010332 >> not using I-node (on process 0) routines >> >> > From xy2102 at columbia.edu Mon Aug 3 14:51:41 2009 From: xy2102 at columbia.edu ((Rebecca) Xuefei YUAN) Date: Mon, 03 Aug 2009 15:51:41 -0400 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? In-Reply-To: References: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> <4ED6AE3A-BEDC-4CC1-8650-D0F0A0CE3678@mcs.anl.gov> <20090803143251.59r3y2mcqo0wg0cs@cubmail.cc.columbia.edu> Message-ID: <20090803155141.z1v7emjl2ossk84g@cubmail.cc.columbia.edu> Dear Barry, I might not state the problem clearly, I saved my solution to a binary file named "solution", so after running the code, there two files "solution" and "solution.info" were formed. Then I would like to the following matlab function to load the data: fid = fopen('solution','r'); c = fread(fid); What I would like to have for c is a 1d Array with 36 elements(real number), however, it turns out that c is a 1d array has 1116 elements(integer), what is wrong with it? Do I miss anything? Or the binary file from Petsc is different from the normal binary file? Thanks very much! R Quoting Barry Smith : > > I use PetscBinaryRead.m then call the plot routines directly in Matlab. > > You can also use the PetscReadBinaryMatlab() stuff to put the contour > commands directly in the .info file and then use PetscReadBinaryMatlab.m > > Barry > > > You can use PetscReadBinary > On Aug 3, 2009, at 1:32 PM, (Rebecca) Xuefei YUAN wrote: > >> Dear Barry, >> >> Thanks very much! If this is the case, do you have any suggestions >> to load the binary file into matlab thus doing the output, like >> contour(u)? >> >> Thanks! >> >> R >> >> Quoting Barry Smith : >> >>> >>> Thanks for reporting this. The documentation for this Matlab >>> function is wrong, it doesn't actually return anything. >>> >>> I have fixed the docs and it will be fixed in the next patch. >>> >>> Barry >>> >>> On Aug 3, 2009, at 8:04 AM, (Rebecca) Xuefei YUAN wrote: >>> >>>> >>>> Thanks, >>>> >>>> -- >>>> (Rebecca) Xuefei YUAN >>>> Department of Applied Physics and Applied Mathematics >>>> Columbia University >>>> Tel:917-399-8032 >>>> www.columbia.edu/~xy2102 >>>> >> >> >> >> -- >> (Rebecca) Xuefei YUAN >> Department of Applied Physics and Applied Mathematics >> Columbia University >> Tel:917-399-8032 >> www.columbia.edu/~xy2102 >> -- (Rebecca) Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From bsmith at mcs.anl.gov Mon Aug 3 15:10:14 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 3 Aug 2009 15:10:14 -0500 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? In-Reply-To: <20090803155141.z1v7emjl2ossk84g@cubmail.cc.columbia.edu> References: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> <4ED6AE3A-BEDC-4CC1-8650-D0F0A0CE3678@mcs.anl.gov> <20090803143251.59r3y2mcqo0wg0cs@cubmail.cc.columbia.edu> <20090803155141.z1v7emjl2ossk84g@cubmail.cc.columbia.edu> Message-ID: Use bin/matlab/PetscBinaryRead.m with c = PetscBinaryRead('solution'); The help for PetscBinaryRead.m indicates other uses also, for example, if you put multiple PETSc objects into the same viewer. Barry On Aug 3, 2009, at 2:51 PM, (Rebecca) Xuefei YUAN wrote: > Dear Barry, > > I might not state the problem clearly, I saved my solution to a > binary file named "solution", so after running the code, there two > files "solution" and "solution.info" were formed. Then I would like > to the following matlab function to load the data: > > fid = fopen('solution','r'); > c = fread(fid); > > What I would like to have for c is a 1d Array with 36 elements(real > number), however, it turns out that c is a 1d array has 1116 > elements(integer), what is wrong with it? Do I miss anything? Or the > binary file from Petsc is different from the normal binary file? > > Thanks very much! > > R > > Quoting Barry Smith : > >> >> I use PetscBinaryRead.m then call the plot routines directly in >> Matlab. >> >> You can also use the PetscReadBinaryMatlab() stuff to put the >> contour >> commands directly in the .info file and then use >> PetscReadBinaryMatlab.m >> >> Barry >> >> >> You can use PetscReadBinary >> On Aug 3, 2009, at 1:32 PM, (Rebecca) Xuefei YUAN wrote: >> >>> Dear Barry, >>> >>> Thanks very much! If this is the case, do you have any >>> suggestions to load the binary file into matlab thus doing the >>> output, like contour(u)? >>> >>> Thanks! >>> >>> R >>> >>> Quoting Barry Smith : >>> >>>> >>>> Thanks for reporting this. The documentation for this Matlab >>>> function is wrong, it doesn't actually return anything. >>>> >>>> I have fixed the docs and it will be fixed in the next patch. >>>> >>>> Barry >>>> >>>> On Aug 3, 2009, at 8:04 AM, (Rebecca) Xuefei YUAN wrote: >>>> >>>>> >>>>> Thanks, >>>>> >>>>> -- >>>>> (Rebecca) Xuefei YUAN >>>>> Department of Applied Physics and Applied Mathematics >>>>> Columbia University >>>>> Tel:917-399-8032 >>>>> www.columbia.edu/~xy2102 >>>>> >>> >>> >>> >>> -- >>> (Rebecca) Xuefei YUAN >>> Department of Applied Physics and Applied Mathematics >>> Columbia University >>> Tel:917-399-8032 >>> www.columbia.edu/~xy2102 >>> > > > > -- > (Rebecca) Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > From xy2102 at columbia.edu Mon Aug 3 15:16:47 2009 From: xy2102 at columbia.edu ((Rebecca) Xuefei YUAN) Date: Mon, 03 Aug 2009 16:16:47 -0400 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? In-Reply-To: References: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> <4ED6AE3A-BEDC-4CC1-8650-D0F0A0CE3678@mcs.anl.gov> <20090803143251.59r3y2mcqo0wg0cs@cubmail.cc.columbia.edu> <20090803155141.z1v7emjl2ossk84g@cubmail.cc.columbia.edu> Message-ID: <20090803161647.ldird6fvms4c0so0@cubmail.cc.columbia.edu> I used PetscBinaryRead('solution') before, but it kept returning the error message: -------------------------------------------------------------- >> c = PetscBinaryRead('solution'); ??? Undefined function or method 'freader' for input arguments of type 'char'. Error in ==> PetscBinaryRead at 22 fd = freader(inarg); ------------------------------------------------------------- The Petsc function I use to save this binary file is PetscViewerBinaryOpen(). Since that is not successful, I started on my own... Cheers, R Quoting Barry Smith : > > Use bin/matlab/PetscBinaryRead.m with > > c = PetscBinaryRead('solution'); > > The help for PetscBinaryRead.m indicates other uses also, for > example, if you put multiple PETSc objects into the same viewer. > > Barry > > On Aug 3, 2009, at 2:51 PM, (Rebecca) Xuefei YUAN wrote: > >> Dear Barry, >> >> I might not state the problem clearly, I saved my solution to a >> binary file named "solution", so after running the code, there two >> files "solution" and "solution.info" were formed. Then I would like >> to the following matlab function to load the data: >> >> fid = fopen('solution','r'); >> c = fread(fid); >> >> What I would like to have for c is a 1d Array with 36 elements(real >> number), however, it turns out that c is a 1d array has 1116 >> elements(integer), what is wrong with it? Do I miss anything? Or >> the binary file from Petsc is different from the normal binary file? >> >> Thanks very much! >> >> R >> >> Quoting Barry Smith : >> >>> >>> I use PetscBinaryRead.m then call the plot routines directly in Matlab. >>> >>> You can also use the PetscReadBinaryMatlab() stuff to put the contour >>> commands directly in the .info file and then use PetscReadBinaryMatlab.m >>> >>> Barry >>> >>> >>> You can use PetscReadBinary >>> On Aug 3, 2009, at 1:32 PM, (Rebecca) Xuefei YUAN wrote: >>> >>>> Dear Barry, >>>> >>>> Thanks very much! If this is the case, do you have any >>>> suggestions to load the binary file into matlab thus doing the >>>> output, like contour(u)? >>>> >>>> Thanks! >>>> >>>> R >>>> >>>> Quoting Barry Smith : >>>> >>>>> >>>>> Thanks for reporting this. The documentation for this Matlab >>>>> function is wrong, it doesn't actually return anything. >>>>> >>>>> I have fixed the docs and it will be fixed in the next patch. >>>>> >>>>> Barry >>>>> >>>>> On Aug 3, 2009, at 8:04 AM, (Rebecca) Xuefei YUAN wrote: >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> (Rebecca) Xuefei YUAN >>>>>> Department of Applied Physics and Applied Mathematics >>>>>> Columbia University >>>>>> Tel:917-399-8032 >>>>>> www.columbia.edu/~xy2102 >>>>>> >>>> >>>> >>>> >>>> -- >>>> (Rebecca) Xuefei YUAN >>>> Department of Applied Physics and Applied Mathematics >>>> Columbia University >>>> Tel:917-399-8032 >>>> www.columbia.edu/~xy2102 >>>> >> >> >> >> -- >> (Rebecca) Xuefei YUAN >> Department of Applied Physics and Applied Mathematics >> Columbia University >> Tel:917-399-8032 >> www.columbia.edu/~xy2102 >> -- (Rebecca) Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From bsmith at mcs.anl.gov Mon Aug 3 15:21:00 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 3 Aug 2009 15:21:00 -0500 Subject: There is no return value for bin/matlab/PetscReadBinaryMatlab.m? In-Reply-To: <20090803161647.ldird6fvms4c0so0@cubmail.cc.columbia.edu> References: <20090803090456.9ju68fcqw0gsowso@cubmail.cc.columbia.edu> <4ED6AE3A-BEDC-4CC1-8650-D0F0A0CE3678@mcs.anl.gov> <20090803143251.59r3y2mcqo0wg0cs@cubmail.cc.columbia.edu> <20090803155141.z1v7emjl2ossk84g@cubmail.cc.columbia.edu> <20090803161647.ldird6fvms4c0so0@cubmail.cc.columbia.edu> Message-ID: Do you have the latest PETSc 3.0.0 with all its patches? On Aug 3, 2009, at 3:16 PM, (Rebecca) Xuefei YUAN wrote: > I used PetscBinaryRead('solution') before, but it kept returning the > error message: > -------------------------------------------------------------- >>> c = PetscBinaryRead('solution'); > ??? Undefined function or method 'freader' for input arguments of > type 'char'. > > Error in ==> PetscBinaryRead at 22 > fd = freader(inarg); > ------------------------------------------------------------- > The Petsc function I use to save this binary file is > PetscViewerBinaryOpen(). > > Since that is not successful, I started on my own... > > Cheers, > > R > > > Quoting Barry Smith : > >> >> Use bin/matlab/PetscBinaryRead.m with >> >> c = PetscBinaryRead('solution'); >> >> The help for PetscBinaryRead.m indicates other uses also, for >> example, if you put multiple PETSc objects into the same viewer. >> >> Barry >> >> On Aug 3, 2009, at 2:51 PM, (Rebecca) Xuefei YUAN wrote: >> >>> Dear Barry, >>> >>> I might not state the problem clearly, I saved my solution to a >>> binary file named "solution", so after running the code, there >>> two files "solution" and "solution.info" were formed. Then I >>> would like to the following matlab function to load the data: >>> >>> fid = fopen('solution','r'); >>> c = fread(fid); >>> >>> What I would like to have for c is a 1d Array with 36 >>> elements(real number), however, it turns out that c is a 1d array >>> has 1116 elements(integer), what is wrong with it? Do I miss >>> anything? Or the binary file from Petsc is different from the >>> normal binary file? >>> >>> Thanks very much! >>> >>> R >>> >>> Quoting Barry Smith : >>> >>>> >>>> I use PetscBinaryRead.m then call the plot routines directly in >>>> Matlab. >>>> >>>> You can also use the PetscReadBinaryMatlab() stuff to put the >>>> contour >>>> commands directly in the .info file and then use >>>> PetscReadBinaryMatlab.m >>>> >>>> Barry >>>> >>>> >>>> You can use PetscReadBinary >>>> On Aug 3, 2009, at 1:32 PM, (Rebecca) Xuefei YUAN wrote: >>>> >>>>> Dear Barry, >>>>> >>>>> Thanks very much! If this is the case, do you have any >>>>> suggestions to load the binary file into matlab thus doing the >>>>> output, like contour(u)? >>>>> >>>>> Thanks! >>>>> >>>>> R >>>>> >>>>> Quoting Barry Smith : >>>>> >>>>>> >>>>>> Thanks for reporting this. The documentation for this Matlab >>>>>> function is wrong, it doesn't actually return anything. >>>>>> >>>>>> I have fixed the docs and it will be fixed in the next patch. >>>>>> >>>>>> Barry >>>>>> >>>>>> On Aug 3, 2009, at 8:04 AM, (Rebecca) Xuefei YUAN wrote: >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -- >>>>>>> (Rebecca) Xuefei YUAN >>>>>>> Department of Applied Physics and Applied Mathematics >>>>>>> Columbia University >>>>>>> Tel:917-399-8032 >>>>>>> www.columbia.edu/~xy2102 >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> (Rebecca) Xuefei YUAN >>>>> Department of Applied Physics and Applied Mathematics >>>>> Columbia University >>>>> Tel:917-399-8032 >>>>> www.columbia.edu/~xy2102 >>>>> >>> >>> >>> >>> -- >>> (Rebecca) Xuefei YUAN >>> Department of Applied Physics and Applied Mathematics >>> Columbia University >>> Tel:917-399-8032 >>> www.columbia.edu/~xy2102 >>> > > > > -- > (Rebecca) Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > From xy2102 at columbia.edu Mon Aug 3 16:54:43 2009 From: xy2102 at columbia.edu ((Rebecca) Xuefei YUAN) Date: Mon, 03 Aug 2009 17:54:43 -0400 Subject: configure error Message-ID: <20090803175443.wyklsh82gw0wwcgc@cubmail.cc.columbia.edu> Hi, I have the following error but I do not understand.. rebecca at YuanWork:~/soft/petsc-3.0.0-p7$ ./config/configure.py --with-cc=gcc --download-f-blas-lapack=1 --download-mpich=/home/rebecca/soft/petsc-3.0.0-p7/externalpackages/mpich-1.2.7p1/ --with-shared=0 --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions ================================================================================= Configuring PETSc to compile on your system ================================================================================= ********************************************************************************* ERROR in COMMAND LINE ARGUMENT to config/configure.py --------------------------------------------------------------------------------------- Invalid download location: /home/rebecca/soft/petsc-3.0.0-p7/externalpackages/mpich-1.2.7p1/ for key download-mpich ********************************************************************************* THanks, -- (Rebecca) Xuefei YUAN Department of Applied Physics and Applied Mathematics Columbia University Tel:917-399-8032 www.columbia.edu/~xy2102 From bsmith at mcs.anl.gov Mon Aug 3 16:56:59 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 3 Aug 2009 16:56:59 -0500 Subject: configure error In-Reply-To: <20090803175443.wyklsh82gw0wwcgc@cubmail.cc.columbia.edu> References: <20090803175443.wyklsh82gw0wwcgc@cubmail.cc.columbia.edu> Message-ID: <3A9140BC-F0E0-4AE3-918D-D6C118B7169A@mcs.anl.gov> Just list --download-mpich you don't need to list the directory. Barry On Aug 3, 2009, at 4:54 PM, (Rebecca) Xuefei YUAN wrote: > Hi, > > I have the following error but I do not understand.. > > > rebecca at YuanWork:~/soft/petsc-3.0.0-p7$ ./config/configure.py --with- > cc=gcc --download-f-blas-lapack=1 --download-mpich=/home/rebecca/ > soft/petsc-3.0.0-p7/externalpackages/mpich-1.2.7p1/ --with-shared=0 > --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions > = > = > = > = > = > = > = > = > = > = > = > ====================================================================== > Configuring PETSc to compile on your system > = > = > = > = > = > = > = > = > = > = > = > ====================================================================== > ********************************************************************************* > ERROR in COMMAND LINE ARGUMENT to config/configure.py > --------------------------------------------------------------------------------------- > Invalid download location: /home/rebecca/soft/petsc-3.0.0-p7/ > externalpackages/mpich-1.2.7p1/ for key download-mpich > ********************************************************************************* > > > THanks, > > > > -- > (Rebecca) Xuefei YUAN > Department of Applied Physics and Applied Mathematics > Columbia University > Tel:917-399-8032 > www.columbia.edu/~xy2102 > From u.tabak at tudelft.nl Tue Aug 4 09:11:50 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Tue, 04 Aug 2009 16:11:50 +0200 Subject: Slepc interface problem, IPNorm: The inner product is not well defined! Message-ID: <4A7841A6.6070607@tudelft.nl> Dear all, I have a generalized eigenvalue problem where B is a singular matrix(rather more problematic there are some columns and rows with zeros due to a finite element modelling issue.). I read section 3.4.4 of slepc manual on the purification, however i could not understand much about the details. But I guess, I should use as problem type, EPS_PGNHEP, however, I am still getting a "IPNorm: The inner product is not well defined!" error. Can someone/developers shed some light on this? Best regards, Umut From enjoywm at cs.wm.edu Tue Aug 4 15:51:52 2009 From: enjoywm at cs.wm.edu (Yixun Liu) Date: Tue, 04 Aug 2009 16:51:52 -0400 Subject: make test errors Message-ID: <4A789F68.1040408@cs.wm.edu> Hi, I encounter the error as I run make test, >make test Running test examples to verify correct installation Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: dat_registry_add_provider I list the output of configure. Hope it helpful for diagnosis. > ./config/configure.py --download-parmetis *************************************************************************************************************************************************************** TESTING: alternateConfigureLibrary from PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) Compilers: C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g Linkers: Static linker: /usr/bin/ar cr PETSc: ** ** Before running "make" your PETSC_ARCH must be specified with: ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) ** PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 ** ** Now build the libraries with "make all" ** Clanguage: C PETSc shared libraries: disabled PETSc dynamic libraries: disabled Scalar type:real MPI: Includes: -I/usr/lib64/mpi/gcc/openmpi/include -I/usr/lib64/mpi/gcc/openmpi/lib64 X11: Includes: [''] Library: ['-lX11'] BLAS/LAPACK: -llapack -lblas ParMetis: Includes: -I/usr/lib64/mpi/gcc/openmpi/include -I/usr/lib64/mpi/gcc/openmpi/lib64 -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include Library: -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis -lmetis ******************************************************************************************************************************************************************* Yixun From knepley at gmail.com Tue Aug 4 15:54:02 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Aug 2009 15:54:02 -0500 Subject: make test errors In-Reply-To: <4A789F68.1040408@cs.wm.edu> References: <4A789F68.1040408@cs.wm.edu> Message-ID: On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu wrote: > Hi, > I encounter the error as I run make test, > >make test > Running test examples to verify correct installation > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > See > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider This looks like a bizarre failure with your OpenMPI compiler wrappers. Do they work at all? Matt > > I list the output of configure. Hope it helpful for diagnosis. > > > ./config/configure.py --download-parmetis > > *************************************************************************************************************************************************************** > TESTING: alternateConfigureLibrary from > PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > Compilers: > C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 > Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > Linkers: > Static linker: /usr/bin/ar cr > PETSc: > ** > ** Before running "make" your PETSC_ARCH must be specified with: > ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > ** > PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > ** > ** Now build the libraries with "make all" > ** > Clanguage: C > PETSc shared libraries: disabled > PETSc dynamic libraries: disabled > Scalar type:real > MPI: > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > -I/usr/lib64/mpi/gcc/openmpi/lib64 > X11: > Includes: [''] > Library: ['-lX11'] > BLAS/LAPACK: -llapack -lblas > ParMetis: > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > -I/usr/lib64/mpi/gcc/openmpi/lib64 > -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > Library: > -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis > -lmetis > > ******************************************************************************************************************************************************************* > > Yixun > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From enjoywm at cs.wm.edu Tue Aug 4 16:03:00 2009 From: enjoywm at cs.wm.edu (Yixun Liu) Date: Tue, 04 Aug 2009 17:03:00 -0400 Subject: make test errors In-Reply-To: References: <4A789F68.1040408@cs.wm.edu> Message-ID: <4A78A204.2050701@cs.wm.edu> I'm not familiar with MPI. Are there some easy way to test it? Matthew Knepley wrote: > On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu > wrote: > > Hi, > I encounter the error as I run make test, > >make test > Running test examples to verify correct installation > Possible error running C/C++ src/snes/examples/tutorials/ex19 with > 1 MPI > process > See > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined > symbol: > dat_registry_add_provider > > > This looks like a bizarre failure with your OpenMPI compiler wrappers. > Do they work > at all? > > Matt > > > > I list the output of configure. Hope it helpful for diagnosis. > > > ./config/configure.py --download-parmetis > *************************************************************************************************************************************************************** > TESTING: alternateConfigureLibrary from > PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > Compilers: > C Compiler: mpicc -Wall -Wwrite-strings > -Wno-strict-aliasing -g3 > Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > Linkers: > Static linker: /usr/bin/ar cr > PETSc: > ** > ** Before running "make" your PETSC_ARCH must be specified with: > ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > ** > PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > ** > ** Now build the libraries with "make all" > ** > Clanguage: C > PETSc shared libraries: disabled > PETSc dynamic libraries: disabled > Scalar type:real > MPI: > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > -I/usr/lib64/mpi/gcc/openmpi/lib64 > X11: > Includes: [''] > Library: ['-lX11'] > BLAS/LAPACK: -llapack -lblas > ParMetis: > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > -I/usr/lib64/mpi/gcc/openmpi/lib64 > -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > Library: > -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis > -lmetis > ******************************************************************************************************************************************************************* > > Yixun > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From balay at mcs.anl.gov Tue Aug 4 16:06:57 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 4 Aug 2009 16:06:57 -0500 (CDT) Subject: make test errors In-Reply-To: <4A78A204.2050701@cs.wm.edu> References: <4A789F68.1040408@cs.wm.edu> <4A78A204.2050701@cs.wm.edu> Message-ID: Try running manually - and see if the example works. These messages could be verbose - and misleading. i.e cd src/ksp/ksp/examples/tutorials make ex2 mpiexec -n 2 ./ex2 Or you can avoid this by installing mpich - instead of openmpi. Satish On Tue, 4 Aug 2009, Yixun Liu wrote: > I'm not familiar with MPI. Are there some easy way to test it? > Matthew Knepley wrote: > > On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu > > wrote: > > > > Hi, > > I encounter the error as I run make test, > > >make test > > Running test examples to verify correct installation > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with > > 1 MPI > > process > > See > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined > > symbol: > > dat_registry_add_provider > > > > > > This looks like a bizarre failure with your OpenMPI compiler wrappers. > > Do they work > > at all? > > > > Matt > > > > > > > > I list the output of configure. Hope it helpful for diagnosis. > > > > > ./config/configure.py --download-parmetis > > *************************************************************************************************************************************************************** > > TESTING: alternateConfigureLibrary from > > PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > > Compilers: > > C Compiler: mpicc -Wall -Wwrite-strings > > -Wno-strict-aliasing -g3 > > Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > > Linkers: > > Static linker: /usr/bin/ar cr > > PETSc: > > ** > > ** Before running "make" your PETSC_ARCH must be specified with: > > ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > > ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > > ** > > PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > > ** > > ** Now build the libraries with "make all" > > ** > > Clanguage: C > > PETSc shared libraries: disabled > > PETSc dynamic libraries: disabled > > Scalar type:real > > MPI: > > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > > -I/usr/lib64/mpi/gcc/openmpi/lib64 > > X11: > > Includes: [''] > > Library: ['-lX11'] > > BLAS/LAPACK: -llapack -lblas > > ParMetis: > > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > > -I/usr/lib64/mpi/gcc/openmpi/lib64 > > -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > > Library: > > -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > > -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis > > -lmetis > > ******************************************************************************************************************************************************************* > > > > Yixun > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > From knepley at gmail.com Tue Aug 4 16:21:58 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Aug 2009 16:21:58 -0500 Subject: make test errors In-Reply-To: <4A78A204.2050701@cs.wm.edu> References: <4A789F68.1040408@cs.wm.edu> <4A78A204.2050701@cs.wm.edu> Message-ID: On Tue, Aug 4, 2009 at 4:03 PM, Yixun Liu wrote: > I'm not familiar with MPI. Are there some easy way to test it? 1) You can tryu using MPICH instead, with --download-mpich 2) You can try compiling a simple program on your own and running it 3) I suspect that there may be a problem running on this machine. It might be that the configuration is different than the "compute nodes" since it appears that you can't see a library you need. Matt > > Matthew Knepley wrote: > > On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu > > wrote: > > > > Hi, > > I encounter the error as I run make test, > > >make test > > Running test examples to verify correct installation > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with > > 1 MPI > > process > > See > > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined > > symbol: > > dat_registry_add_provider > > > > > > This looks like a bizarre failure with your OpenMPI compiler wrappers. > > Do they work > > at all? > > > > Matt > > > > > > > > I list the output of configure. Hope it helpful for diagnosis. > > > > > ./config/configure.py --download-parmetis > > > *************************************************************************************************************************************************************** > > TESTING: alternateConfigureLibrary from > > PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > > Compilers: > > C Compiler: mpicc -Wall -Wwrite-strings > > -Wno-strict-aliasing -g3 > > Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > > Linkers: > > Static linker: /usr/bin/ar cr > > PETSc: > > ** > > ** Before running "make" your PETSC_ARCH must be specified with: > > ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > > ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > > ** > > PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > > ** > > ** Now build the libraries with "make all" > > ** > > Clanguage: C > > PETSc shared libraries: disabled > > PETSc dynamic libraries: disabled > > Scalar type:real > > MPI: > > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > > -I/usr/lib64/mpi/gcc/openmpi/lib64 > > X11: > > Includes: [''] > > Library: ['-lX11'] > > BLAS/LAPACK: -llapack -lblas > > ParMetis: > > Includes: -I/usr/lib64/mpi/gcc/openmpi/include > > -I/usr/lib64/mpi/gcc/openmpi/lib64 > > -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > > Library: > > -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > > -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis > > -lmetis > > > ******************************************************************************************************************************************************************* > > > > Yixun > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Aug 4 19:03:58 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 4 Aug 2009 19:03:58 -0500 Subject: Compiling petsc-dev with c++/boost/sieve In-Reply-To: <20090730150811.GG23977@tchpc.tcd.ie> References: <20090730150811.GG23977@tchpc.tcd.ie> Message-ID: On Thu, Jul 30, 2009 at 10:08 AM, wrote: > Hi, > > I'm trying to run examples from the > $PETSC_DIR/src/dm/mesh/examples/tutorials directory. I'm interested > in the mixedpoisson example, but I've tried compiling the ex[1-3] > examples, with results that I give below. I've also compiled some > files in the $PETSC_DIR/src/dm/mesh/examples/tests directory with > mostly failures as well. Details of the petsc compilation are at the > bottom of the email I really apologize for taking so long to answer. I have been traveling, and have been up against several deadlines. The short answer is that this is my fault. The package is moving very fast because we have few users, and thus are willing to make lots of changes for them (which are hard to keep up to date in the examples). So 1) If you look on the website at http://www.mcs.anl.gov/petsc/petsc-as/documentation/tutorials/index.html and get the slides from the Short Course in China (at the top), this has a very detailed, step-by- step construction of the Bratu equation for general elements. The repository that goes with the slides is http://petsc.cs.iit.edu/petsc/GUCAS09TutorialCode/ Each checkin adds a little more functionality, until you have a full simulation. Your comments would be appreciated. 2) I know you mentioned the mixed-poisson demo. I will get one working as soon as I can. If you start before me and have questions or need help, don't hesitate to ask. 3) There is a specific list for these questions, sieve-dev at mcs.anl.gov. Thanks, Matt > > These compilations take place with PETSC_DIR=, > and with PETSC_ARCH set, but I've had similar problems after a 'make > install' > > Examples elsewhere in the petsc tree appear to compile and run > correctly > > I don't want to waste anyones time wading through reams of output, so > I'm really asking whether this output indicates obviously that I have > failed to configure/compile petsc correctly with c++/boost/sieve, and > therefore have no hope of successfully compiling the examples? > > Darach > > petsc-dev: HG revision: f9c1b044f127006244143f415f257bfbc93c7a6e HG Date: > Tue Jul 28 16:59:48 2009 -0500 > > %> gcc --version > gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-44) > ..... > > %> make ex1 > mpicxx -o ex1.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -g > -I/home/user/Compile/petsc-dev/src/dm/mesh/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -I/home/user/Compile/petsc-dev/linux-gnu-cxx-debug/include > -I/home/user/Compile/petsc-dev/include > -I/home/user/Compile/petsc-dev/include/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -D__SDIR__="src/dm/mesh/examples/tutorials/" ex1.c > ex1.c: In function ? PetscErrorCode CreatePartition(_p_Mesh*, > _p_SectionInt**)? : > ex1.c:170: error: invalid initialization of reference of type ? > ALE::Obj ALE::malloc_allocator > > >, > ALE::malloc_allocator ALE::malloc_allocator > > > > >&? from > expression of type ? ALE::Obj > >? > /home/user/Compile/petsc-dev/include/petscmesh.h:108: error: in passing > argument 2 of ? PetscErrorCode MeshGetMesh(_p_Mesh*, > ALE::Obj ALE::malloc_allocator > > >, > ALE::malloc_allocator ALE::malloc_allocator > > > > >&)? > ex1.c:171: error: invalid conversion from ? int? to ? const char*? > ex1.c:171: error: invalid conversion from ? _p_SectionInt**? to ? > PetscInt? > /home/user/Compile/petsc-dev/include/petscmesh.h:255: error: too few > arguments to function ? PetscErrorCode MeshGetCellSectionInt(_p_Mesh*, > const char*, PetscInt, _p_SectionInt**)? > ex1.c:171: error: at this point in file > ex1.c:173: error: invalid initialization of reference of type ? > ALE::Obj >, > ALE::malloc_allocator ALE::malloc_allocator > > >&? from expression of type ? > ALE::Obj, > ALE::UniformSection > > >, ALE::malloc_allocator, > ALE::UniformSection > > > > >? > .... > > > %> make ex2 > mpicxx -o ex2.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -g > -I/home/user/Compile/petsc-dev/src/dm/mesh/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -I/home/user/Compile/petsc-dev/linux-gnu-cxx-debug/include > -I/home/user/Compile/petsc-dev/include > -I/home/user/Compile/petsc-dev/include/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -D__SDIR__="src/dm/mesh/examples/tutorials/" ex2.c > ex2.c:22:38: error: ../src/dm/mesh/meshpcice.h: No such file or directory > ex2.c: In function ? PetscErrorCode CreateSquareBoundary(const > ALE::Obj >&)? : > ex2.c:148: error: ? class ALE::Mesh::topology_type? has not been > declared > ex2.c:148: error: expected initializer before ? patch? > ex2.c:153: error: ? topology_type? is not a member of ? ALE::Mesh? > ex2.c:153: error: ? topology_type? is not a member of ? ALE::Mesh? > ex2.c:153: error: template argument 1 is invalid > ex2.c:153: error: template argument 2 is invalid > ex2.c:153: error: invalid type in declaration before ? =? token > ex2.c:153: error: expected type-specifier > ex2.c:153: error: invalid conversion from ? int*? to ? int? > ex2.c:153: error: expected ? ,? or ? ;? > ex2.c:182: error: base operand of ? ->? is not a pointer > ex2.c:182: error: ? patch? was not declared in this scope > ex2.c:183: error: base operand of ? ->? is not a pointer > ex2.c:184: error: ? class ALE::Mesh? has no member named ? setTopology? > ex2.c:185: error: ? SieveBuilder? is not a member of ? ALE::New? > ex2.c:185: error: expected primary-expression before ? >? token > ex2.c:185: error: ? ::buildCoordinates? has not been declared > ex2.c:187: error: ? topology_type? is not a member of ? ALE::Mesh? > ex2.c:187: error: ? topology_type? is not a member of ? ALE::Mesh? > ex2.c:187: error: template argument 1 is invalid > .... > > %> make ex3 > mpicxx -o ex3.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -g > -I/home/user/Compile/petsc-dev/src/dm/mesh/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -I/home/user/Compile/petsc-dev/linux-gnu-cxx-debug/include > -I/home/user/Compile/petsc-dev/include > -I/home/user/Compile/petsc-dev/include/sieve > -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > -D__SDIR__="src/dm/mesh/examples/tutorials/" ex3.c > ex3.c:26: error: ? Two? is not a member of ? ALE? > ex3.c:26: error: ? Two? is not a member of ? ALE? > ex3.c:26: error: template argument 1 is invalid > ex3.c:26: error: template argument 2 is invalid > ex3.c:27: error: ? Two? is not a member of ? ALE? > ex3.c:27: error: ? Two? is not a member of ? ALE? > .... > > > Compilation Details: > --------------------- > [user at machine petsc-dev]$ ./config/configure.py > --prefix=/home/user/install_home/petsc-dev-defaultboost-sieve-comp > --with-scalar-type=complex --with-clanguage=cxx --with-boost=1 > --download-boost=/home/user/Compile/petsc-dev/externalpackages/boost.tar.gz > --with-sieve=1 > > ================================================================================= > Configuring PETSc to compile on your system > > ================================================================================= > TESTING: alternateConfigureLibrary from > PETSc.packages.mpi4py(config/PETSc/packages/mpi4py.py:54) > Compilers: > C Compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -g3 > C++ Compiler: mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -g > Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > Linkers: > Static linker: /usr/bin/ar cr > PETSc: > PETSC_ARCH: linux-gnu-cxx-debug > PETSC_DIR: /home/user/Compile/petsc-dev > ** > ** Now build the libraries with "make all" > ** > Clanguage: Cxx > Scalar type:complex > MPI: > Includes: -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > X11: > Includes: [''] > Library: ['-lX11'] > PETSc shared libraries: disabled > PETSc dynamic libraries: disabled > BLAS/LAPACK: -llapack -lblas > Sieve: > Includes: -I/home/user/Compile/petsc-dev/include/sieve > Boost: > Includes: -I/home/user/Compile/petsc-dev/externalpackages/Boost/ > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/include > -I/misc/shared/apps/openmpi/gcc/64/1.2.8/lib > c2html: > sowing: > > > > Using mpiexec: /misc/shared/apps/openmpi/gcc/64/1.2.8/bin/mpiexec > ========================================== > /bin/rm -f -f > /home/user/Compile/petsc-dev/linux-gnu-cxx-debug/lib/libpetsc*.* > /bin/rm -f -f > /home/user/Compile/petsc-dev/linux-gnu-cxx-debug/include/petsc*.mod > BEGINNING TO COMPILE LIBRARIES IN ALL DIRECTORIES > ========================================= > .... > libfast in: /home/user/Compile/petsc-dev/src/snes/examples/tutorials > libfast in: /home/user/Compile/petsc-dev/src/snes/examples/tutorials/ex10d > libfast in: /home/user/Compile/petsc-dev/src/snes/utils > libfast in: /home/user/Compile/petsc-dev/src/snes/utils/sieve > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1159: instantiated > from ? ALE::IFSieve::~IFSieve() [with Point_ = int, > Allocator_ = ALE::malloc_allocator]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1348: instantiated > from ? ALE::IBundle ArrowSection_>::~IBundle() [with Sieve_ = ALE::IFSieve ALE::malloc_allocator >, RealSection_ = ALE::IGeneralSection double, ALE::malloc_allocator >, IntSection_ = > ALE::IGeneralSection >, Label_ = > ALE::LabelSifter ALE::malloc_allocator > >, ArrowSection_ > = ALE::UniformSection, int, 1, > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1576: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IMesh ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IMesh int, ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > meshmgsnes.c:63: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > meshmgsnes.c:349: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > .... > libfast in: /home/user/Compile/petsc-dev/src/dm/da/utils/f90-custom > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1148: instantiated > from ? ALE::IFSieve::IFSieve(ompi_communicator_t*, int) > [with Point_ = int, Allocator_ = ALE::malloc_allocator]? > mesh.c:1642: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > mesh.c:2846: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > meshpcice.c:387: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > meshpcice.c:388: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1159: instantiated > from ? ALE::IFSieve::~IFSieve() [with Point_ = int, > Allocator_ = ALE::malloc_allocator]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1348: instantiated > from ? ALE::IBundle ArrowSection_>::~IBundle() [with Sieve_ = ALE::IFSieve ALE::malloc_allocator >, RealSection_ = ALE::IGeneralSection double, ALE::malloc_allocator >, IntSection_ = > ALE::IGeneralSection >, Label_ = > ALE::LabelSifter ALE::malloc_allocator > >, ArrowSection_ > = ALE::UniformSection, int, 1, > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1576: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IMesh ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IMesh int, ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > meshpflotran.c:235: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > meshpflotran.c:903: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1159: instantiated > from ? ALE::IFSieve::~IFSieve() [with Point_ = int, > Allocator_ = ALE::malloc_allocator]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1332: instantiated > from ? ALE::IBundle ArrowSection_>::IBundle(ompi_communicator_t*, int) [with Sieve_ = > ALE::IFSieve >, RealSection_ = > ALE::IGeneralSection >, > IntSection_ = ALE::IGeneralSection >, > Label_ = ALE::LabelSifter ALE::malloc_allocator > >, ArrowSection_ > = ALE::UniformSection, int, 1, > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1602: instantiated > from ? ALE::IMesh::IMesh(ompi_communicator_t*, int, int) [with > Label_ = ALE::LabelSifter ALE::malloc_allocator > >]? > meshexodus.c:183: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > meshexodus.c:364: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > section.c: In function ? PetscErrorCode > SectionRealCreateLocalVector(_p_SectionReal*, _p_Vec**)? : > section.c:582: warning: unused variable ? ierr? > /home/user/Compile/petsc-dev/include/sieve/Field.hh: In member function ? > void ALE::GeneralSection::zero() > [with Point_ = int, Value_ = int, Alloc_ = ALE::malloc_allocator, > Atlas_ = ALE::IUniformSection ALE::malloc_allocator >, BCAtlas_ = ALE::ISection ALE::malloc_allocator >]? : > section.c:1354: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/Field.hh:1740: warning: passing > ? double? for argument 1 to ? void ALE::GeneralSection Alloc_, Atlas_, BCAtlas_>::set(Value_) [with Point_ = int, Value_ = int, > Alloc_ = ALE::malloc_allocator, Atlas_ = ALE::IUniformSection ALE::Point, 1, ALE::malloc_allocator >, BCAtlas_ = > ALE::ISection >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: At global scope: > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1159: instantiated > from ? ALE::IFSieve::~IFSieve() [with Point_ = int, > Allocator_ = ALE::malloc_allocator]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/petscmesh_viewers.hh:490: > instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > section.c:1532: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/sieve > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/impls > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/impls/cartesian > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence? : > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieveDef::Sequence, A = > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:1159: instantiated > from ? ALE::IFSieve::~IFSieve() [with Point_ = int, > Allocator_ = ALE::malloc_allocator]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:799: instantiated > from ? void ALE::Obj::destroy() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IFSieve ALE::malloc_allocator >, A = ALE::malloc_allocator ALE::malloc_allocator > >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1348: instantiated > from ? ALE::IBundle ArrowSection_>::~IBundle() [with Sieve_ = ALE::IFSieve ALE::malloc_allocator >, RealSection_ = ALE::IGeneralSection double, ALE::malloc_allocator >, IntSection_ = > ALE::IGeneralSection >, Label_ = > ALE::LabelSifter ALE::malloc_allocator > >, ArrowSection_ > = ALE::UniformSection, int, 1, > ALE::malloc_allocator >]? > /home/user/Compile/petsc-dev/include/sieve/Mesh.hh:1576: instantiated > from ? void ALE::Obj::destroy() [with X = > ALE::IMesh ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > /home/user/Compile/petsc-dev/include/sieve/ALE_mem.hh:745: instantiated > from ? ALE::Obj::~Obj() [with X = ALE::IMesh int, ALE::malloc_allocator > > >, A = > ALE::malloc_allocator ALE::malloc_allocator > > > >]? > cartesian.c:263: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:957: warning: ? class > ALE::IFSieveDef::Sequence? has virtual functions but non-virtual > destructor > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh: In instantiation of ? > ALE::IFSieveDef::Sequence::const_iterator? : > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:994: instantiated > from ? ALE::IFSieveDef::Sequence::const_iterator > ALE::IFSieveDef::Sequence::begin() const [with PointType_ = > int]? > cartesian.c:269: instantiated from here > /home/user/Compile/petsc-dev/include/sieve/ISieve.hh:960: warning: ? class > ALE::IFSieveDef::Sequence::const_iterator? has virtual functions but > non-virtual destructor > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/ftn-auto > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/ftn-custom > libfast in: /home/user/Compile/petsc-dev/src/dm/mesh/f90-custom > libfast in: /home/user/Compile/petsc-dev/src/dm/adda > libfast in: /home/user/Compile/petsc-dev/src/dm/adda/examples > libfast in: /home/user/Compile/petsc-dev/src/dm/adda/examples/tests > libfast in: /home/user/Compile/petsc-dev/src/dm/adda/ftn-auto > libfast in: /home/user/Compile/petsc-dev/src/dm/f90-mod > libfast in: /home/user/Compile/petsc-dev/src/dm/ftn-custom > libfast in: /home/user/Compile/petsc-dev/src/contrib > libfast in: /home/user/Compile/petsc-dev/src/contrib/fun3d > libfast in: /home/user/Compile/petsc-dev/src/benchmarks > libfast in: /home/user/Compile/petsc-dev/src/docs > libfast in: /home/user/Compile/petsc-dev/include > libfast in: /home/user/Compile/petsc-dev/include/finclude > libfast in: /home/user/Compile/petsc-dev/include/finclude/ftn-auto > libfast in: /home/user/Compile/petsc-dev/include/finclude/ftn-custom > libfast in: /home/user/Compile/petsc-dev/include/private > libfast in: /home/user/Compile/petsc-dev/include/sieve > libfast in: /home/user/Compile/petsc-dev/include/ftn-auto > petschf.c: In function ? PetscErrorCode petscmemcpy_(void*, void*, > size_t*, int*)? : > petschf.c:42: warning: no return statement in function returning non-void > libfast in: /home/user/Compile/petsc-dev/tutorials > libfast in: /home/user/Compile/petsc-dev/tutorials/multiphysics > Completed building libraries > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From enjoywm at cs.wm.edu Wed Aug 5 08:35:35 2009 From: enjoywm at cs.wm.edu (Yixun Liu) Date: Wed, 05 Aug 2009 09:35:35 -0400 Subject: make test errors In-Reply-To: References: <4A789F68.1040408@cs.wm.edu> <4A78A204.2050701@cs.wm.edu> Message-ID: <4A798AA7.80108@cs.wm.edu> I did as what you said, mpiexec -n 2 ./ex2 DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: dat_registry_add_provider DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: dat_registry_add_provider WARNING: Failed to open "OpenIB-cma" [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. This may be a real error or it may be an invalid entry in the uDAPL Registry which is contained in the dat.conf file. Contact your local System Administrator to confirm the availability of the interfaces in the dat.conf file. Satish Balay wrote: > Try running manually - and see if the example works. These messages > could be verbose - and misleading. > > i.e > > cd src/ksp/ksp/examples/tutorials > make ex2 > mpiexec -n 2 ./ex2 > > Or you can avoid this by installing mpich - instead of openmpi. > > Satish > > On Tue, 4 Aug 2009, Yixun Liu wrote: > > >> I'm not familiar with MPI. Are there some easy way to test it? >> Matthew Knepley wrote: >> >>> On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu >> > wrote: >>> >>> Hi, >>> I encounter the error as I run make test, >>> >make test >>> Running test examples to verify correct installation >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with >>> 1 MPI >>> process >>> See >>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html >>> DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined >>> symbol: >>> dat_registry_add_provider >>> >>> >>> This looks like a bizarre failure with your OpenMPI compiler wrappers. >>> Do they work >>> at all? >>> >>> Matt >>> >>> >>> >>> I list the output of configure. Hope it helpful for diagnosis. >>> >>> > ./config/configure.py --download-parmetis >>> *************************************************************************************************************************************************************** >>> TESTING: alternateConfigureLibrary from >>> PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) >>> Compilers: >>> C Compiler: mpicc -Wall -Wwrite-strings >>> -Wno-strict-aliasing -g3 >>> Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g >>> Linkers: >>> Static linker: /usr/bin/ar cr >>> PETSc: >>> ** >>> ** Before running "make" your PETSC_ARCH must be specified with: >>> ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) >>> ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) >>> ** >>> PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 >>> ** >>> ** Now build the libraries with "make all" >>> ** >>> Clanguage: C >>> PETSc shared libraries: disabled >>> PETSc dynamic libraries: disabled >>> Scalar type:real >>> MPI: >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 >>> X11: >>> Includes: [''] >>> Library: ['-lX11'] >>> BLAS/LAPACK: -llapack -lblas >>> ParMetis: >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 >>> -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include >>> Library: >>> -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib >>> -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis >>> -lmetis >>> ******************************************************************************************************************************************************************* >>> >>> Yixun >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >>> >> > > From knepley at gmail.com Wed Aug 5 08:39:24 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 5 Aug 2009 08:39:24 -0500 Subject: make test errors In-Reply-To: <4A798AA7.80108@cs.wm.edu> References: <4A789F68.1040408@cs.wm.edu> <4A78A204.2050701@cs.wm.edu> <4A798AA7.80108@cs.wm.edu> Message-ID: On Wed, Aug 5, 2009 at 8:35 AM, Yixun Liu wrote: > I did as what you said, > mpiexec -n 2 ./ex2 > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. As it says, you should probably contact your local admin. It does not seem to have much to do with PETSc. Matt > > Satish Balay wrote: > > Try running manually - and see if the example works. These messages > > could be verbose - and misleading. > > > > i.e > > > > cd src/ksp/ksp/examples/tutorials > > make ex2 > > mpiexec -n 2 ./ex2 > > > > Or you can avoid this by installing mpich - instead of openmpi. > > > > Satish > > > > On Tue, 4 Aug 2009, Yixun Liu wrote: > > > > > >> I'm not familiar with MPI. Are there some easy way to test it? > >> Matthew Knepley wrote: > >> > >>> On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu >>> > wrote: > >>> > >>> Hi, > >>> I encounter the error as I run make test, > >>> >make test > >>> Running test examples to verify correct installation > >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with > >>> 1 MPI > >>> process > >>> See > >>> > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > >>> DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined > >>> symbol: > >>> dat_registry_add_provider > >>> > >>> > >>> This looks like a bizarre failure with your OpenMPI compiler wrappers. > >>> Do they work > >>> at all? > >>> > >>> Matt > >>> > >>> > >>> > >>> I list the output of configure. Hope it helpful for diagnosis. > >>> > >>> > ./config/configure.py --download-parmetis > >>> > *************************************************************************************************************************************************************** > >>> TESTING: alternateConfigureLibrary from > >>> PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > >>> Compilers: > >>> C Compiler: mpicc -Wall -Wwrite-strings > >>> -Wno-strict-aliasing -g3 > >>> Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > >>> Linkers: > >>> Static linker: /usr/bin/ar cr > >>> PETSc: > >>> ** > >>> ** Before running "make" your PETSC_ARCH must be specified with: > >>> ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > >>> ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > >>> ** > >>> PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > >>> ** > >>> ** Now build the libraries with "make all" > >>> ** > >>> Clanguage: C > >>> PETSc shared libraries: disabled > >>> PETSc dynamic libraries: disabled > >>> Scalar type:real > >>> MPI: > >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include > >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 > >>> X11: > >>> Includes: [''] > >>> Library: ['-lX11'] > >>> BLAS/LAPACK: -llapack -lblas > >>> ParMetis: > >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include > >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 > >>> -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > >>> Library: > >>> -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > >>> -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > -lparmetis > >>> -lmetis > >>> > ******************************************************************************************************************************************************************* > >>> > >>> Yixun > >>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > >>> their experiments lead. > >>> -- Norbert Wiener > >>> > >> > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 5 08:40:06 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 5 Aug 2009 08:40:06 -0500 (CDT) Subject: make test errors In-Reply-To: <4A798AA7.80108@cs.wm.edu> References: <4A789F68.1040408@cs.wm.edu> <4A78A204.2050701@cs.wm.edu> <4A798AA7.80108@cs.wm.edu> Message-ID: You would have to check with your sysadmin - who installed this openMPI on how to correctly use it. Satish On Wed, 5 Aug 2009, Yixun Liu wrote: > I did as what you said, > mpiexec -n 2 ./ex2 > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined symbol: > dat_registry_add_provider > > WARNING: Failed to open "OpenIB-cma" > [DAT_PROVIDER_NOT_FOUND:DAT_NAME_NOT_REGISTERED]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > > > Satish Balay wrote: > > Try running manually - and see if the example works. These messages > > could be verbose - and misleading. > > > > i.e > > > > cd src/ksp/ksp/examples/tutorials > > make ex2 > > mpiexec -n 2 ./ex2 > > > > Or you can avoid this by installing mpich - instead of openmpi. > > > > Satish > > > > On Tue, 4 Aug 2009, Yixun Liu wrote: > > > > > >> I'm not familiar with MPI. Are there some easy way to test it? > >> Matthew Knepley wrote: > >> > >>> On Tue, Aug 4, 2009 at 3:51 PM, Yixun Liu >>> > wrote: > >>> > >>> Hi, > >>> I encounter the error as I run make test, > >>> >make test > >>> Running test examples to verify correct installation > >>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with > >>> 1 MPI > >>> process > >>> See > >>> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html > >>> DAT: library load failure: /usr/lib64/libdaplcma.so.1: undefined > >>> symbol: > >>> dat_registry_add_provider > >>> > >>> > >>> This looks like a bizarre failure with your OpenMPI compiler wrappers. > >>> Do they work > >>> at all? > >>> > >>> Matt > >>> > >>> > >>> > >>> I list the output of configure. Hope it helpful for diagnosis. > >>> > >>> > ./config/configure.py --download-parmetis > >>> *************************************************************************************************************************************************************** > >>> TESTING: alternateConfigureLibrary from > >>> PETSc.packages.petsc4py(config/PETSc/packages/petsc4py.py:69) > >>> Compilers: > >>> C Compiler: mpicc -Wall -Wwrite-strings > >>> -Wno-strict-aliasing -g3 > >>> Fortran Compiler: mpif90 -Wall -Wno-unused-variable -g > >>> Linkers: > >>> Static linker: /usr/bin/ar cr > >>> PETSc: > >>> ** > >>> ** Before running "make" your PETSC_ARCH must be specified with: > >>> ** ** setenv PETSC_ARCH linux-gnu-c-debug (csh/tcsh) > >>> ** ** PETSC_ARCH=linux-gnu-c-debug; export PETSC_ARCH (sh/bash) > >>> ** > >>> PETSC_DIR: /home/scratch/yixun/petsc-3.0.0-p3 > >>> ** > >>> ** Now build the libraries with "make all" > >>> ** > >>> Clanguage: C > >>> PETSc shared libraries: disabled > >>> PETSc dynamic libraries: disabled > >>> Scalar type:real > >>> MPI: > >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include > >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 > >>> X11: > >>> Includes: [''] > >>> Library: ['-lX11'] > >>> BLAS/LAPACK: -llapack -lblas > >>> ParMetis: > >>> Includes: -I/usr/lib64/mpi/gcc/openmpi/include > >>> -I/usr/lib64/mpi/gcc/openmpi/lib64 > >>> -I/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/include > >>> Library: > >>> -Wl,-rpath,/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib > >>> -L/home/scratch/yixun/petsc-3.0.0-p3/linux-gnu-c-debug/lib -lparmetis > >>> -lmetis > >>> ******************************************************************************************************************************************************************* > >>> > >>> Yixun > >>> > >>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which > >>> their experiments lead. > >>> -- Norbert Wiener > >>> > >> > > > > > > From u.tabak at tudelft.nl Thu Aug 6 05:05:16 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 06 Aug 2009 12:05:16 +0200 Subject: A point on documentation of MatSetValuesBlocked Message-ID: <4A7AAADC.7090800@tudelft.nl> Dear all, There is sth which took some time for me to figure out from the documentation http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBlocked.html#MatSetValuesBlocked Or my English is getting worser, In the explanations that are a part of this function, The m and n count the NUMBER of blocks in the row direction and column direction, NOT the total number of rows/columns; for example, if the block size is 2 and you are passing in values for rows 2,3,4,5 then m would be 2 (not 4). The values in idxm would be 1 2; that is the first index for each block divided by the block size . But the puzzling point is that there are no pointers for the setting of the block size and hyperlinks on size also do not direct to the right locations in my humble opinion. However, proper block size setting should be done before using this routine with MatSetBlockSize. Am I right? And this page has no pointers to that function. After reading the documetation, I had the idea in the mind that, you decide on the block size, and do the computations on paper for m,idxm,n and idxn to be supplied to the routine but ,apparently, without a call to the function the programs can not know the block size, which is logical. I felt myself a bit dump, and wanted to notify you about this if someone else also experiences the same problem in the future, maybe adding a link there for the above mentioned function is a good idea. //MatSetBlockSize(A, 3); const int idxmNew1[] = {0}; const int idxnNew1[] = {0}; PetscScalar vals1[] = { 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0 }; MatSetValuesBlocked(A, 1, idxmNew1, 1, idxnNew1, vals1, INSERT_VALUES); ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); Running the above piece of code of course sets only M[0][0] with out proper setting of the blocks. Or do I still have some misunderstanding on the use of this function? Best regards, Umut From knepley at gmail.com Thu Aug 6 07:59:55 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Aug 2009 07:59:55 -0500 Subject: A point on documentation of MatSetValuesBlocked In-Reply-To: <4A7AAADC.7090800@tudelft.nl> References: <4A7AAADC.7090800@tudelft.nl> Message-ID: I have pushed a fix for the documentation that explains the need for MatSetBlockSize(). It will appear in the next patch release, and the next nightly build of the online documentation. Thanks, Matt On Thu, Aug 6, 2009 at 5:05 AM, Umut Tabak wrote: > Dear all, > > There is sth which took some time for me to figure out from the > documentation > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBlocked.html#MatSetValuesBlocked > > Or my English is getting worser, In the explanations that are a part of > this function, > > The m and n count the NUMBER of blocks in the row direction and column > direction, NOT the total number of rows/columns; for example, if the block > size < > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/size.html#size> > is 2 and you are passing in values for rows 2,3,4,5 then m would be 2 (not > 4). The values in idxm would be 1 2; that is the first index for each block > divided by the block size < > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Sys/size.html#size>. > > > But the puzzling point is that there are no pointers for the setting of the > block size and hyperlinks on size also do not direct to the right locations > in my humble opinion. > > However, proper block size setting should be done before using this routine > with MatSetBlockSize. Am I right? > > And this page has no pointers to that function. After reading the > documetation, I had the idea in the mind that, you decide on the block size, > and do the computations on paper for m,idxm,n and idxn to be supplied to the > routine but ,apparently, without a call to the function the programs can not > know the block size, which is logical. > > I felt myself a bit dump, and wanted to notify you about this if someone > else also experiences the same problem in the future, maybe adding a link > there for the above mentioned function is a good idea. > > //MatSetBlockSize(A, 3); > const int idxmNew1[] = {0}; > const int idxnNew1[] = {0}; > PetscScalar vals1[] = { 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0 }; > MatSetValuesBlocked(A, 1, idxmNew1, 1, idxnNew1, vals1, INSERT_VALUES); > ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr); > > Running the above piece of code of course sets only M[0][0] with out proper > setting of the blocks. Or do I still have some misunderstanding on the use > of this function? > > Best regards, > Umut > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Thu Aug 6 08:18:17 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 06 Aug 2009 15:18:17 +0200 Subject: efficient matrix block assignment Message-ID: <4A7AD819.3040802@tudelft.nl> Dear all, I am trying some block matrix assembly in Petsc, so far, I was puzzling with the right calling and set up of MatSetValuesBlocked. Matthew Knepley confirmed my previous post. I guess it is the function I should use. But it accepts the values as a 1D array, which is represented in row major order(by default). So the burden on this approach is to convert my Matrix into a 1D array, this seemed as the only solution for the moment. If I would like to use this function. Another way I could think of by reading the documentation of Matrices, is to use MatGetColumnVector and assign over column vectors, but the above approach is far more efficient, I guess. BTW, some blocks are square and some are rectangular, with block size type operations, I guess it is not possible to assign rectangular matrices to some part of a larger square matrix. Are there any other efficient suggestions? Best regards, Umut From knepley at gmail.com Thu Aug 6 08:29:45 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Aug 2009 08:29:45 -0500 Subject: efficient matrix block assignment In-Reply-To: <4A7AD819.3040802@tudelft.nl> References: <4A7AD819.3040802@tudelft.nl> Message-ID: On Thu, Aug 6, 2009 at 8:18 AM, Umut Tabak wrote: > Dear all, > > I am trying some block matrix assembly in Petsc, so far, I was puzzling > with the right calling and set up of MatSetValuesBlocked. Matthew Knepley > confirmed my previous post. I guess it is the function I should use. But it > accepts the values as a 1D array, which is represented in row major > order(by default). So the burden on this approach is to convert my Matrix > into a 1D array, this seemed as the only solution for the moment. If I would > like to use this function. I need to understand what you mean here. The 1D array we accept here is the input values. It is physically 1D, but conceptually 2D, and by default is in row-major order. This has nothing to do with your larger Mat object. > > Another way I could think of by reading the documentation of Matrices, is > to use MatGetColumnVector and assign over column vectors, but the above > approach is far more efficient, I guess. You cannot assign that way, only through MatSetValues*(). > > BTW, some blocks are square and some are rectangular, with block size type > operations, I guess it is not possible to assign rectangular matrices to > some part of a larger square matrix. Not true. You can have any mxn dimension to the input values. I think there is some underlying confusion here that I do not understand. I would suggest you look at the example in the manual. Matt > > Are there any other efficient suggestions? > > Best regards, > Umut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Thu Aug 6 08:55:31 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 06 Aug 2009 15:55:31 +0200 Subject: efficient matrix block assignment In-Reply-To: References: <4A7AD819.3040802@tudelft.nl> Message-ID: <4A7AE0D3.1000607@tudelft.nl> Matthew Knepley wrote: > On Thu, Aug 6, 2009 at 8:18 AM, Umut Tabak > wrote: > > Dear all, > > I am trying some block matrix assembly in Petsc, so far, I was > puzzling with the right calling and set up of MatSetValuesBlocked. > Matthew Knepley confirmed my previous post. I guess it is the > function I should use. But it accepts the values as a 1D array, > which is represented in row major order(by default). So the > burden on this approach is to convert my Matrix into a 1D array, > this seemed as the only solution for the moment. If I would like > to use this function. > > > I need to understand what you mean here. The 1D array we accept here > is the input values. It is physically 1D, but conceptually > 2D, and by default is in row-major order. This has nothing to do with > your larger Mat object. Precisely, what i meant, there is a 2d array which is conceptually a matrix in my case(where I read from a Finite element code). So I must represent it as a 1D array. > > > > BTW, some blocks are square and some are rectangular, with block > size type operations, I guess it is not possible to assign > rectangular matrices to some part of a larger square matrix. > > > Not true. You can have any mxn dimension to the input values. I think > there is some underlying confusion here > that I do not understand. I would suggest you look at the example in > the manual. > > Matt This is also confusing for me for the moment. But there is only one parameter, for operation MatSetBlockSize(Mat, blocksize), this was what I meant by "block size" above. I should check the docs more, I guess. Hope made it clear. Thanks Umut From bsmith at mcs.anl.gov Thu Aug 6 11:37:21 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 6 Aug 2009 11:37:21 -0500 Subject: efficient matrix block assignment In-Reply-To: <4A7AE0D3.1000607@tudelft.nl> References: <4A7AD819.3040802@tudelft.nl> <4A7AE0D3.1000607@tudelft.nl> Message-ID: <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> On Aug 6, 2009, at 8:55 AM, Umut Tabak wrote: > Matthew Knepley wrote: >> On Thu, Aug 6, 2009 at 8:18 AM, Umut Tabak > >> wrote: >> >> Dear all, >> >> I am trying some block matrix assembly in Petsc, so far, I was >> puzzling with the right calling and set up of MatSetValuesBlocked. >> Matthew Knepley confirmed my previous post. I guess it is the >> function I should use. But it accepts the values as a 1D array, >> which is represented in row major order(by default). So the >> burden on this approach is to convert my Matrix into a 1D array, >> this seemed as the only solution for the moment. If I would like >> to use this function. >> >> >> I need to understand what you mean here. The 1D array we accept >> here is the input values. It is physically 1D, but conceptually >> 2D, and by default is in row-major order. This has nothing to do >> with your larger Mat object. > Precisely, what i meant, there is a 2d array which is conceptually a > matrix in my case(where I read from a Finite element code). So I > must represent it as a 1D array. I most cases (Fortran always) a 2d array is just stored in memory as a 1d array so no conversion is needed. In C if you declare an array double something[3][3] then yes it is not suitable to directly passing into MatSetValues....(). >> >> >> BTW, some blocks are square and some are rectangular, with block >> size type operations, I guess it is not possible to assign >> rectangular matrices to some part of a larger square matrix. >> >> >> Not true. You can have any mxn dimension to the input values. I >> think there is some underlying confusion here >> that I do not understand. I would suggest you look at the example >> in the manual. >> >> Matt > This is also confusing for me for the moment. But there is only one > parameter, for operation MatSetBlockSize(Mat, blocksize), this was > what I meant by "block size" above. I should check the docs more, I > guess. The "blocks" in the matrix are always square. When you call MatSetValues() you can put in a rectangular block of values. When you call MatSetValuesBlock() you can put in a rectangular set of blocks. Barry > > Hope made it clear. > Thanks > Umut > From u.tabak at tudelft.nl Thu Aug 6 12:29:24 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 06 Aug 2009 19:29:24 +0200 Subject: efficient matrix block assignment In-Reply-To: <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> References: <4A7AD819.3040802@tudelft.nl> <4A7AE0D3.1000607@tudelft.nl> <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> Message-ID: <4A7B12F4.7010807@tudelft.nl> Barry Smith wrote: > > On Aug 6, 2009, at 8:55 AM, Umut Tabak wrote: > > The "blocks" in the matrix are always square. When you call > MatSetValues() you can put in a rectangular block of values. When you > call MatSetValuesBlock() you can put in a rectangular set of blocks. > > Barry Hi Barry, The blocks issue is clear from my tries. But you use two successive "can" for rectangular matrices, I guess the second can is "can not" for MatSetValuesBlock(), right? Thanks for the clarifications, Umut From knepley at gmail.com Thu Aug 6 12:45:35 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 6 Aug 2009 12:45:35 -0500 Subject: efficient matrix block assignment In-Reply-To: <4A7B12F4.7010807@tudelft.nl> References: <4A7AD819.3040802@tudelft.nl> <4A7AE0D3.1000607@tudelft.nl> <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> <4A7B12F4.7010807@tudelft.nl> Message-ID: On Thu, Aug 6, 2009 at 12:29 PM, Umut Tabak wrote: > Barry Smith wrote: > >> >> On Aug 6, 2009, at 8:55 AM, Umut Tabak wrote: >> >> The "blocks" in the matrix are always square. When you call >> MatSetValues() you can put in a rectangular block of values. When you call >> MatSetValuesBlock() you can put in a rectangular set of blocks. >> >> Barry >> > Hi Barry, > > The blocks issue is clear from my tries. But you use two successive "can" > for rectangular matrices, I guess the second can is "can not" for > MatSetValuesBlock(), right? No, both "cans" are correct. You can do both things. Matt > > Thanks for the clarifications, > Umut > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Thu Aug 6 16:27:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 6 Aug 2009 18:27:33 -0300 Subject: efficient matrix block assignment In-Reply-To: References: <4A7AD819.3040802@tudelft.nl> <4A7AE0D3.1000607@tudelft.nl> <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> <4A7B12F4.7010807@tudelft.nl> Message-ID: >> Barry Smith wrote: >>> MatSetValuesBlock() you can put in a rectangular set of blocks. >>> Indeed. As barry pointed out, you can put a rectangular m x n *set* of *blocks*... For example: array([[0, 0, 1, 1, 2, 2], [0, 0, 1, 1, 2, 2], [3, 3, 4, 4, 5, 5], [3, 3, 4, 4, 5, 5]]) is a 2x3 set of blocks where each (square) block has bs=2, i.e. each block is 2x2 square. > >> >> Thanks for the clarifications, >> Umut > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From u.tabak at tudelft.nl Thu Aug 6 16:37:35 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 06 Aug 2009 23:37:35 +0200 Subject: efficient matrix block assignment In-Reply-To: References: <4A7AD819.3040802@tudelft.nl> <4A7AE0D3.1000607@tudelft.nl> <2D9E18EF-F856-4979-A316-CEEA1A15EDC1@mcs.anl.gov> <4A7B12F4.7010807@tudelft.nl> Message-ID: <4A7B4D1F.9010302@tudelft.nl> > Indeed. As barry pointed out, you can put a rectangular m x n *set* of > *blocks*... > > For example: > > array([[0, 0, 1, 1, 2, 2], > [0, 0, 1, 1, 2, 2], > [3, 3, 4, 4, 5, 5], > [3, 3, 4, 4, 5, 5]]) > > is a 2x3 set of blocks where each (square) block has bs=2, i.e. each > block is 2x2 square. > Was trying to read Barry's email again because I missed that *set* of *blocks*... and to sketch on paper some matrices, right on time arrived your email ;) , thanks for the example. Best, Umut From irfan.khan at gatech.edu Fri Aug 7 18:17:54 2009 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Fri, 7 Aug 2009 19:17:54 -0400 (EDT) Subject: PETSC_LIB In-Reply-To: <1743702503.659471249686819343.JavaMail.root@mail8.gatech.edu> Message-ID: <220856782.659871249687074671.JavaMail.root@mail8.gatech.edu> Hi I have been compiling and running my Petsc code on a TERAGRID cluster (Steele - Purdue) for sometime now. Recently they made changes to their operating system and also installed a clean version of Petsc (it now makes use of MPICH-2 instead of MPICH). However my make file is not successfully linking anymore. The error message I get is as follows. ld: cannot find -llapack make: *** [te] Error 1 I see that the blaslapack package is not available under "externalpackages" directory. I presume the link error is because the lapack library is included in the PETSC_LIB command and this library may not have been installed during PETSc installation. I say this because the make file works fine on my laptop where I have installed blas-lapack during PETSc installation. Please correct me if I am wrong. How can I resolve this link error? Kindly suggest. Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) From knepley at gmail.com Fri Aug 7 18:37:06 2009 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 7 Aug 2009 18:37:06 -0500 Subject: PETSC_LIB In-Reply-To: <220856782.659871249687074671.JavaMail.root@mail8.gatech.edu> References: <1743702503.659471249686819343.JavaMail.root@mail8.gatech.edu> <220856782.659871249687074671.JavaMail.root@mail8.gatech.edu> Message-ID: The easiest way to link correctly is to use the PETSc makefile format: prog: file1.o file2.o ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} Matt On Fri, Aug 7, 2009 at 6:17 PM, wrote: > Hi > I have been compiling and running my Petsc code on a TERAGRID cluster > (Steele - Purdue) for sometime now. Recently they made changes to their > operating system and also installed a clean version of Petsc (it now makes > use of MPICH-2 instead of MPICH). > > However my make file is not successfully linking anymore. The error message > I get is as follows. > > ld: cannot find -llapack > make: *** [te] Error 1 > > I see that the blaslapack package is not available under "externalpackages" > directory. I presume the link error is because the lapack library is > included in the PETSC_LIB command and this library may not have been > installed during PETSc installation. I say this because the make file works > fine on my laptop where I have installed blas-lapack during PETSc > installation. Please correct me if I am wrong. > > How can I resolve this link error? Kindly suggest. > > > Thank you > Irfan > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Aug 9 05:55:39 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 09 Aug 2009 12:55:39 +0200 Subject: memory leak from valgrind on ex1 Message-ID: <4A7EAB2B.8090403@tudelft.nl> Dear all, I was trying to locate my memory-leaking problems due to a class I wrote in combination with Petsc, during these checks I used valgrind in the very basic modes, but there is a point I could not understand about Petsc programs. I used the example which is the in the "getting started" section of the manual(${PETSC_DIR}/src/ksp/ksp/examples/tutorials/ex1.c). A linear system solution. Using valgrind on this problem reported some "definitely lost" parts. I could not understand this error. Can someone help me on understanding this point, I am not a pro, but want to become a user of Petsc by trying to understand the main parts of the library? I used "valgrind --leak-check=yes ./ex1" Best regards, Umut From knepley at gmail.com Sun Aug 9 08:52:54 2009 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 9 Aug 2009 08:52:54 -0500 Subject: memory leak from valgrind on ex1 In-Reply-To: <4A7EAB2B.8090403@tudelft.nl> References: <4A7EAB2B.8090403@tudelft.nl> Message-ID: There are some strings which might not get freed, but they are very small. How big are we talking? Matt On Sun, Aug 9, 2009 at 5:55 AM, Umut Tabak wrote: > Dear all, > > I was trying to locate my memory-leaking problems due to a class I wrote in > combination with Petsc, during these checks I used valgrind in the very > basic modes, but there is a point I could not understand about Petsc > programs. I used the example which is the in the "getting started" section > of the manual(${PETSC_DIR}/src/ksp/ksp/examples/tutorials/ex1.c). > > A linear system solution. Using valgrind on this problem reported some > "definitely lost" parts. I could not understand this error. Can someone help > me on understanding this point, I am not a pro, but want to become a user of > Petsc by trying to understand the main parts of the library? > > I used "valgrind --leak-check=yes ./ex1" > > Best regards, > Umut > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Sun Aug 9 09:09:20 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Sun, 09 Aug 2009 16:09:20 +0200 Subject: memory leak from valgrind on ex1 In-Reply-To: References: <4A7EAB2B.8090403@tudelft.nl> Message-ID: <4A7ED890.2020200@tudelft.nl> Matthew Knepley wrote: > There are some strings which might not get freed, but they are very > small. How big are we talking? > > Matt > Hi, Thanks for the quick return on Sunday, Here is the result of ex1.c that is mentioned in the previous mail, they are small indeed, but I was wondering if there were still things I could not manage with my pointer allocations/assignments, that is the reason why I made a post. I also attached another valgrind output, This was just to be sure that I am coding without leaks. BTW, the program seems to work but there are other problems related to eigenvalues I would like to solve for the moment... Best regards, Umut ==23836== Memcheck, a memory error detector. ==23836== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al. ==23836== Using LibVEX rev 1884, a library for dynamic binary translation. ==23836== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP. ==23836== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation framework. ==23836== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al. ==23836== For more details, rerun with: -v ==23836== KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-07, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: jacobi linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=10, cols=10 total: nonzeros=28, allocated nonzeros=50 not using I-node routines Norm of error < 1.e-12, Iterations 5 ==23836== ==23836== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 26 from 3) ==23836== malloc/free: in use at exit: 292 bytes in 11 blocks. ==23836== malloc/free: 1,011 allocs, 1,000 frees, 974,088 bytes allocated. ==23836== For counts of detected errors, rerun with: -v ==23836== searching for pointers to 11 not-freed blocks. ==23836== checked 1,678,608 bytes. ==23836== ==23836== 292 (52 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 1 of 3 ==23836== at 0x4C278AE: malloc (vg_replace_malloc.c:207) ==23836== by 0x82D410A: (within /lib/libc-2.9.so) ==23836== by 0x82D4A06: __nss_database_lookup (in /lib/libc-2.9.so) ==23836== by 0x918033F: ??? ==23836== by 0x9182264: ??? ==23836== by 0x8282051: getpwuid_r (in /lib/libc-2.9.so) ==23836== by 0x828191E: getpwuid (in /lib/libc-2.9.so) ==23836== by 0x640685B: PetscGetUserName(char*, unsigned long) (fuser.c:68) ==23836== by 0x63ED66E: PetscErrorPrintfInitialize() (errtrace.c:68) ==23836== by 0x644A995: PetscInitialize(int*, char***, char const*, char const*) (pinit.c:521) ==23836== by 0x401645: main (ex1.c:37) ==23836== ==23836== LEAK SUMMARY: ==23836== definitely lost: 52 bytes in 1 blocks. ==23836== indirectly lost: 240 bytes in 10 blocks. ==23836== possibly lost: 0 bytes in 0 blocks. ==23836== still reachable: 0 bytes in 0 blocks. ==23836== suppressed: 0 bytes in 0 blocks. -------------- next part -------------- A non-text attachment was scrubbed... Name: petscVg.dat Type: application/x-ns-proxy-autoconfig Size: 10488 bytes Desc: not available URL: From knepley at gmail.com Sun Aug 9 09:11:54 2009 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 9 Aug 2009 09:11:54 -0500 Subject: memory leak from valgrind on ex1 In-Reply-To: <4A7ED890.2020200@tudelft.nl> References: <4A7EAB2B.8090403@tudelft.nl> <4A7ED890.2020200@tudelft.nl> Message-ID: On Sun, Aug 9, 2009 at 9:09 AM, Umut Tabak wrote: > Matthew Knepley wrote: > >> There are some strings which might not get freed, but they are very small. >> How big are we talking? >> >> Matt >> >> Hi, Thanks for the quick return on Sunday, > > Here is the result of ex1.c that is mentioned in the previous mail, they > are small indeed, but I was wondering if there were still things I could not > manage with my pointer allocations/assignments, that is the reason why I > made a post. These are just the names of some classes which do nto get freed on exit. Matt > > I also attached another valgrind output, This was just to be sure that I am > coding without leaks. BTW, the program seems to work but there are other > problems related to eigenvalues I would like to solve for the moment... > > Best regards, > Umut > > ==23836== Memcheck, a memory error detector. > ==23836== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al. > ==23836== Using LibVEX rev 1884, a library for dynamic binary translation. > ==23836== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP. > ==23836== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation > framework. > ==23836== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al. > ==23836== For more details, rerun with: -v > ==23836== > KSP Object: > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000 > left preconditioning > PC Object: > type: jacobi > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=10, cols=10 > total: nonzeros=28, allocated nonzeros=50 > not using I-node routines > Norm of error < 1.e-12, Iterations 5 > ==23836== > ==23836== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 26 from 3) > ==23836== malloc/free: in use at exit: 292 bytes in 11 blocks. > ==23836== malloc/free: 1,011 allocs, 1,000 frees, 974,088 bytes allocated. > ==23836== For counts of detected errors, rerun with: -v > ==23836== searching for pointers to 11 not-freed blocks. > ==23836== checked 1,678,608 bytes. > ==23836== > ==23836== 292 (52 direct, 240 indirect) bytes in 1 blocks are definitely > lost in loss record 1 of 3 > ==23836== at 0x4C278AE: malloc (vg_replace_malloc.c:207) > ==23836== by 0x82D410A: (within /lib/libc-2.9.so) > ==23836== by 0x82D4A06: __nss_database_lookup (in /lib/libc-2.9.so) > ==23836== by 0x918033F: ??? > ==23836== by 0x9182264: ??? > ==23836== by 0x8282051: getpwuid_r (in /lib/libc-2.9.so) > ==23836== by 0x828191E: getpwuid (in /lib/libc-2.9.so) > ==23836== by 0x640685B: PetscGetUserName(char*, unsigned long) > (fuser.c:68) > ==23836== by 0x63ED66E: PetscErrorPrintfInitialize() (errtrace.c:68) > ==23836== by 0x644A995: PetscInitialize(int*, char***, char const*, char > const*) (pinit.c:521) > ==23836== by 0x401645: main (ex1.c:37) > ==23836== > ==23836== LEAK SUMMARY: > ==23836== definitely lost: 52 bytes in 1 blocks. > ==23836== indirectly lost: 240 bytes in 10 blocks. > ==23836== possibly lost: 0 bytes in 0 blocks. > ==23836== still reachable: 0 bytes in 0 blocks. > ==23836== suppressed: 0 bytes in 0 blocks. > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From irfan.khan at gatech.edu Sun Aug 9 17:33:19 2009 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Sun, 9 Aug 2009 18:33:19 -0400 (EDT) Subject: PETSC_LIB In-Reply-To: <960290591.750051249856939459.JavaMail.root@mail8.gatech.edu> Message-ID: <1647104469.750251249857199333.JavaMail.root@mail8.gatech.edu> Thank for your reply Matt. I have tried to compile the src/dm/ao/examples/tutorials/ex2.c using the makefile provided in same directory. However I received the same error, i.e. ld: cannot find -llapack make: [ex2] Error 1 (ignored) Please note that I was able to compile and run this example successfully before the an update was done on the system. I wanted to get some idea as to the cause of this link error. This way I have something to ask the team at Purdue to look into. Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" To: "PETSc users list" Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern Subject: Re: PETSC_LIB The easiest way to link correctly is to use the PETSc makefile format: prog: file1.o file2.o ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} Matt On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: Hi I have been compiling and running my Petsc code on a TERAGRID cluster (Steele - Purdue) for sometime now. Recently they made changes to their operating system and also installed a clean version of Petsc (it now makes use of MPICH-2 instead of MPICH). However my make file is not successfully linking anymore. The error message I get is as follows. ld: cannot find -llapack make: *** [te] Error 1 I see that the blaslapack package is not available under "externalpackages" directory. I presume the link error is because the lapack library is included in the PETSC_LIB command and this library may not have been installed during PETSc installation. I say this because the make file works fine on my laptop where I have installed blas-lapack during PETSc installation. Please correct me if I am wrong. How can I resolve this link error? Kindly suggest. Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Aug 9 17:36:42 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 9 Aug 2009 17:36:42 -0500 (CDT) Subject: PETSC_LIB In-Reply-To: <1647104469.750251249857199333.JavaMail.root@mail8.gatech.edu> References: <1647104469.750251249857199333.JavaMail.root@mail8.gatech.edu> Message-ID: If the examples don't work [with the provided makefile] - you'll have to check with the person who built this petsc for you.. Perhaps he needs to reinstall petsc - or just install blas/lapack for this to work.. Satish On Sun, 9 Aug 2009, irfan.khan at gatech.edu wrote: > Thank for your reply Matt. I have tried to compile the src/dm/ao/examples/tutorials/ex2.c using the makefile provided in same directory. However I received the same error, i.e. > > ld: cannot find -llapack > make: [ex2] Error 1 (ignored) > > Please note that I was able to compile and run this example successfully before the an update was done on the system. > > I wanted to get some idea as to the cause of this link error. This way I have something to ask the team at Purdue to look into. > > Thank you > Irfan > > > ----- Original Message ----- > From: "Matthew Knepley" > To: "PETSc users list" > Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern > Subject: Re: PETSC_LIB > > The easiest way to link correctly is to use the PETSc makefile format: > > prog: file1.o file2.o > ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} > > Matt > > > On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: > > > Hi > I have been compiling and running my Petsc code on a TERAGRID cluster (Steele - Purdue) for sometime now. Recently they made changes to their operating system and also installed a clean version of Petsc (it now makes use of MPICH-2 instead of MPICH). > > However my make file is not successfully linking anymore. The error message I get is as follows. > > ld: cannot find -llapack > make: *** [te] Error 1 > > I see that the blaslapack package is not available under "externalpackages" directory. I presume the link error is because the lapack library is included in the PETSC_LIB command and this library may not have been installed during PETSc installation. I say this because the make file works fine on my laptop where I have installed blas-lapack during PETSc installation. Please correct me if I am wrong. > > How can I resolve this link error? Kindly suggest. > > > Thank you > Irfan > From irfan.khan at gatech.edu Sun Aug 9 17:55:41 2009 From: irfan.khan at gatech.edu (irfan.khan at gatech.edu) Date: Sun, 9 Aug 2009 18:55:41 -0400 (EDT) Subject: PETSC_LIB In-Reply-To: <1231554539.751381249858362173.JavaMail.root@mail8.gatech.edu> Message-ID: <2090503080.751561249858541841.JavaMail.root@mail8.gatech.edu> Thank you for the helpful suggestions. Suppose blas/lapack are already installed on the cluster, what needs to be done in order for petsc to recognise them without having to reconfigure and reinstall. Thank you Irfan ----- Original Message ----- From: "Satish Balay" To: "PETSc users list" Sent: Sunday, August 9, 2009 6:36:42 PM GMT -05:00 US/Canada Eastern Subject: Re: PETSC_LIB If the examples don't work [with the provided makefile] - you'll have to check with the person who built this petsc for you.. Perhaps he needs to reinstall petsc - or just install blas/lapack for this to work.. Satish On Sun, 9 Aug 2009, irfan.khan at gatech.edu wrote: > Thank for your reply Matt. I have tried to compile the src/dm/ao/examples/tutorials/ex2.c using the makefile provided in same directory. However I received the same error, i.e. > > ld: cannot find -llapack > make: [ex2] Error 1 (ignored) > > Please note that I was able to compile and run this example successfully before the an update was done on the system. > > I wanted to get some idea as to the cause of this link error. This way I have something to ask the team at Purdue to look into. > > Thank you > Irfan > > > ----- Original Message ----- > From: "Matthew Knepley" > To: "PETSc users list" > Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern > Subject: Re: PETSC_LIB > > The easiest way to link correctly is to use the PETSc makefile format: > > prog: file1.o file2.o > ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} > > Matt > > > On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: > > > Hi > I have been compiling and running my Petsc code on a TERAGRID cluster (Steele - Purdue) for sometime now. Recently they made changes to their operating system and also installed a clean version of Petsc (it now makes use of MPICH-2 instead of MPICH). > > However my make file is not successfully linking anymore. The error message I get is as follows. > > ld: cannot find -llapack > make: *** [te] Error 1 > > I see that the blaslapack package is not available under "externalpackages" directory. I presume the link error is because the lapack library is included in the PETSC_LIB command and this library may not have been installed during PETSc installation. I say this because the make file works fine on my laptop where I have installed blas-lapack during PETSc installation. Please correct me if I am wrong. > > How can I resolve this link error? Kindly suggest. > > > Thank you > Irfan > -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) From bsmith at mcs.anl.gov Sun Aug 9 18:11:29 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 9 Aug 2009 18:11:29 -0500 Subject: PETSC_LIB In-Reply-To: <2090503080.751561249858541841.JavaMail.root@mail8.gatech.edu> References: <2090503080.751561249858541841.JavaMail.root@mail8.gatech.edu> Message-ID: <110D8A88-9464-4961-9678-4D4B51906158@mcs.anl.gov> If you know where they are installed then all you need to do is add them to the end of your link line when you make the executable in the makefile; it is not that "PETSc needs to recognize them" but your makefile needs to link against them. BUT: I highly recommend simply reconfiguring and making PETSc to use the latest stuff that was installed on the cluster. Better to install properly then hope that an old install keeps working with a new OS. It should not be difficult to install PETSc for your own use. http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html Relying on someone else to install PETSc for you is much like having someone else brush your teeth for you, possible, but not pleasant nor necessary. Barry On Aug 9, 2009, at 5:55 PM, irfan.khan at gatech.edu wrote: > Thank you for the helpful suggestions. Suppose blas/lapack are > already installed on the cluster, what needs to be done in order for > petsc to recognise them without having to reconfigure and reinstall. > > Thank you > Irfan > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSc users list" > Sent: Sunday, August 9, 2009 6:36:42 PM GMT -05:00 US/Canada Eastern > Subject: Re: PETSC_LIB > > If the examples don't work [with the provided makefile] - you'll have > to check with the person who built this petsc for you.. > > Perhaps he needs to reinstall petsc - or just install blas/lapack for > this to work.. > > Satish > > On Sun, 9 Aug 2009, irfan.khan at gatech.edu wrote: > >> Thank for your reply Matt. I have tried to compile the src/dm/ao/ >> examples/tutorials/ex2.c using the makefile provided in same >> directory. However I received the same error, i.e. >> >> ld: cannot find -llapack >> make: [ex2] Error 1 (ignored) >> >> Please note that I was able to compile and run this example >> successfully before the an update was done on the system. >> >> I wanted to get some idea as to the cause of this link error. This >> way I have something to ask the team at Purdue to look into. >> >> Thank you >> Irfan >> >> >> ----- Original Message ----- >> From: "Matthew Knepley" >> To: "PETSc users list" >> Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern >> Subject: Re: PETSC_LIB >> >> The easiest way to link correctly is to use the PETSc makefile >> format: >> >> prog: file1.o file2.o >> ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} >> >> Matt >> >> >> On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: >> >> >> Hi >> I have been compiling and running my Petsc code on a TERAGRID >> cluster (Steele - Purdue) for sometime now. Recently they made >> changes to their operating system and also installed a clean >> version of Petsc (it now makes use of MPICH-2 instead of MPICH). >> >> However my make file is not successfully linking anymore. The error >> message I get is as follows. >> >> ld: cannot find -llapack >> make: *** [te] Error 1 >> >> I see that the blaslapack package is not available under >> "externalpackages" directory. I presume the link error is because >> the lapack library is included in the PETSC_LIB command and this >> library may not have been installed during PETSc installation. I >> say this because the make file works fine on my laptop where I have >> installed blas-lapack during PETSc installation. Please correct me >> if I am wrong. >> >> How can I resolve this link error? Kindly suggest. >> >> >> Thank you >> Irfan >> > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) From knepley at gmail.com Sun Aug 9 18:12:49 2009 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 9 Aug 2009 18:12:49 -0500 Subject: PETSC_LIB In-Reply-To: <2090503080.751561249858541841.JavaMail.root@mail8.gatech.edu> References: <1231554539.751381249858362173.JavaMail.root@mail8.gatech.edu> <2090503080.751561249858541841.JavaMail.root@mail8.gatech.edu> Message-ID: On Sun, Aug 9, 2009 at 5:55 PM, wrote: > Thank you for the helpful suggestions. Suppose blas/lapack are already > installed on the cluster, what needs to be done in order for petsc to > recognise them without having to reconfigure and reinstall. > Nothing. That is exactly why we have a configure. Matt > Thank you > Irfan > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSc users list" > Sent: Sunday, August 9, 2009 6:36:42 PM GMT -05:00 US/Canada Eastern > Subject: Re: PETSC_LIB > > If the examples don't work [with the provided makefile] - you'll have > to check with the person who built this petsc for you.. > > Perhaps he needs to reinstall petsc - or just install blas/lapack for > this to work.. > > Satish > > On Sun, 9 Aug 2009, irfan.khan at gatech.edu wrote: > > > Thank for your reply Matt. I have tried to compile the > src/dm/ao/examples/tutorials/ex2.c using the makefile provided in same > directory. However I received the same error, i.e. > > > > ld: cannot find -llapack > > make: [ex2] Error 1 (ignored) > > > > Please note that I was able to compile and run this example successfully > before the an update was done on the system. > > > > I wanted to get some idea as to the cause of this link error. This way I > have something to ask the team at Purdue to look into. > > > > Thank you > > Irfan > > > > > > ----- Original Message ----- > > From: "Matthew Knepley" > > To: "PETSc users list" > > Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern > > Subject: Re: PETSC_LIB > > > > The easiest way to link correctly is to use the PETSc makefile format: > > > > prog: file1.o file2.o > > ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} > > > > Matt > > > > > > On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: > > > > > > Hi > > I have been compiling and running my Petsc code on a TERAGRID cluster > (Steele - Purdue) for sometime now. Recently they made changes to their > operating system and also installed a clean version of Petsc (it now makes > use of MPICH-2 instead of MPICH). > > > > However my make file is not successfully linking anymore. The error > message I get is as follows. > > > > ld: cannot find -llapack > > make: *** [te] Error 1 > > > > I see that the blaslapack package is not available under > "externalpackages" directory. I presume the link error is because the lapack > library is included in the PETSC_LIB command and this library may not have > been installed during PETSc installation. I say this because the make file > works fine on my laptop where I have installed blas-lapack during PETSc > installation. Please correct me if I am wrong. > > > > How can I resolve this link error? Kindly suggest. > > > > > > Thank you > > Irfan > > > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From irfan.khan at gatech.edu Sun Aug 9 18:21:37 2009 From: irfan.khan at gatech.edu (Khan, Irfan) Date: Sun, 9 Aug 2009 19:21:37 -0400 (EDT) Subject: PETSC_LIB In-Reply-To: <110D8A88-9464-4961-9678-4D4B51906158@mcs.anl.gov> Message-ID: <489927701.752461249860097256.JavaMail.root@mail8.gatech.edu> "Relying on someone else to install PETSc for you is much like having someone else brush your teeth for you, possible, but not pleasant nor necessary." :-) Thank you Barry. I will try to see if I can install one by myself. Irfan ----- Original Message ----- From: "Barry Smith" To: "PETSc users list" Sent: Sunday, August 9, 2009 7:11:29 PM GMT -05:00 US/Canada Eastern Subject: Re: PETSC_LIB If you know where they are installed then all you need to do is add them to the end of your link line when you make the executable in the makefile; it is not that "PETSc needs to recognize them" but your makefile needs to link against them. BUT: I highly recommend simply reconfiguring and making PETSc to use the latest stuff that was installed on the cluster. Better to install properly then hope that an old install keeps working with a new OS. It should not be difficult to install PETSc for your own use. http://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html Relying on someone else to install PETSc for you is much like having someone else brush your teeth for you, possible, but not pleasant nor necessary. Barry On Aug 9, 2009, at 5:55 PM, irfan.khan at gatech.edu wrote: > Thank you for the helpful suggestions. Suppose blas/lapack are > already installed on the cluster, what needs to be done in order for > petsc to recognise them without having to reconfigure and reinstall. > > Thank you > Irfan > > ----- Original Message ----- > From: "Satish Balay" > To: "PETSc users list" > Sent: Sunday, August 9, 2009 6:36:42 PM GMT -05:00 US/Canada Eastern > Subject: Re: PETSC_LIB > > If the examples don't work [with the provided makefile] - you'll have > to check with the person who built this petsc for you.. > > Perhaps he needs to reinstall petsc - or just install blas/lapack for > this to work.. > > Satish > > On Sun, 9 Aug 2009, irfan.khan at gatech.edu wrote: > >> Thank for your reply Matt. I have tried to compile the src/dm/ao/ >> examples/tutorials/ex2.c using the makefile provided in same >> directory. However I received the same error, i.e. >> >> ld: cannot find -llapack >> make: [ex2] Error 1 (ignored) >> >> Please note that I was able to compile and run this example >> successfully before the an update was done on the system. >> >> I wanted to get some idea as to the cause of this link error. This >> way I have something to ask the team at Purdue to look into. >> >> Thank you >> Irfan >> >> >> ----- Original Message ----- >> From: "Matthew Knepley" >> To: "PETSc users list" >> Sent: Friday, August 7, 2009 7:37:06 PM GMT -05:00 US/Canada Eastern >> Subject: Re: PETSC_LIB >> >> The easiest way to link correctly is to use the PETSc makefile >> format: >> >> prog: file1.o file2.o >> ${CLINKER} -o prog file1.o file2.o ${PETSC_SNES_LIB} >> >> Matt >> >> >> On Fri, Aug 7, 2009 at 6:17 PM, < irfan.khan at gatech.edu > wrote: >> >> >> Hi >> I have been compiling and running my Petsc code on a TERAGRID >> cluster (Steele - Purdue) for sometime now. Recently they made >> changes to their operating system and also installed a clean >> version of Petsc (it now makes use of MPICH-2 instead of MPICH). >> >> However my make file is not successfully linking anymore. The error >> message I get is as follows. >> >> ld: cannot find -llapack >> make: *** [te] Error 1 >> >> I see that the blaslapack package is not available under >> "externalpackages" directory. I presume the link error is because >> the lapack library is included in the PETSC_LIB command and this >> library may not have been installed during PETSc installation. I >> say this because the make file works fine on my laptop where I have >> installed blas-lapack during PETSc installation. Please correct me >> if I am wrong. >> >> How can I resolve this link error? Kindly suggest. >> >> >> Thank you >> Irfan >> > > > -- > PhD Candidate > G.W. Woodruff School of Mechanical Engineering > Georgia Institute of Technology > Atlanta, GA (30332) -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) From wangww2011 at gmail.com Mon Aug 10 07:17:47 2009 From: wangww2011 at gmail.com (Weiwei Wang) Date: Mon, 10 Aug 2009 20:17:47 +0800 Subject: cannot find -lustdio with matlab on Message-ID: <4A800FEB.6060206@gmail.com> Dear All, I want to check the determinant of some sparse matrix in petsc but I didn't know how to do it using petsc directly so I try to save the sparse matrix in Matlab format, then calculate the determinant in Matlab, so I reinstalled petsc with the opinion "--with-matlab-dir=...", but the following error appeared: /usr/bin/ld: cannot find -lustdio collect2: ld returned 1 exit status make[2]: [shared_linux] Error 1 (ignored) I don't know where I can find the lib "ustdio" or other ways to calculate the determinant, the system of my box is debian 5, so any suggestion is appreciated. Thanks, weiwei wang From thomas.witkowski at tu-dresden.de Mon Aug 10 08:28:24 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 10 Aug 2009 15:28:24 +0200 Subject: How to get information about the off-diagonal submatrix Message-ID: <4A802078.4040908@tu-dresden.de> Hi, how is it possible to print just the off-diagonal submatrix of one process of a parallel mpiaij matrix? Thomas From knepley at gmail.com Mon Aug 10 08:53:09 2009 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 10 Aug 2009 08:53:09 -0500 Subject: How to get information about the off-diagonal submatrix In-Reply-To: <4A802078.4040908@tu-dresden.de> References: <4A802078.4040908@tu-dresden.de> Message-ID: If this is just for debugging, the easiest way I think is to call MatGetSubmatrix() with a column IS that omits the local columns. Matt On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Hi, > > how is it possible to print just the off-diagonal submatrix of one process > of a parallel mpiaij matrix? > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Aug 10 09:07:38 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Aug 2009 09:07:38 -0500 (CDT) Subject: How to get information about the off-diagonal submatrix In-Reply-To: References: <4A802078.4040908@tu-dresden.de> Message-ID: alternatively - just access the private datastructe and print.. #include "../src/mat/impls/aij/mpi/mpiaij.h" Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; MatView(mpiaij->B,...); Satish On Mon, 10 Aug 2009, Matthew Knepley wrote: > If this is just for debugging, the easiest way I think is to call > MatGetSubmatrix() with > a column IS that omits the local columns. > > Matt > > On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > > > Hi, > > > > how is it possible to print just the off-diagonal submatrix of one process > > of a parallel mpiaij matrix? > > > > Thomas > > > > > > From hzhang at mcs.anl.gov Mon Aug 10 09:10:47 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Mon, 10 Aug 2009 09:10:47 -0500 (CDT) Subject: cannot find -lustdio with matlab on In-Reply-To: <4A800FEB.6060206@gmail.com> References: <4A800FEB.6060206@gmail.com> Message-ID: weiwei, The easist way is run your petsc code with the option '-mat_view_matlab', and dump it into a file. Then use Matlab to load it for whatever you want to investigate. See petsc user manual on "Using Matlab with PETSc". Hong On Mon, 10 Aug 2009, Weiwei Wang wrote: > Dear All, > I want to check the determinant of some sparse matrix in petsc but I > didn't know how to do it using petsc directly so I try to save the sparse > matrix in Matlab format, then calculate the determinant in Matlab, so I > reinstalled petsc with the opinion "--with-matlab-dir=...", but the following > error appeared: > > /usr/bin/ld: cannot find -lustdio > collect2: ld returned 1 exit status > make[2]: [shared_linux] Error 1 (ignored) > > I don't know where I can find the lib "ustdio" or other ways to calculate the > determinant, the system of my box is debian 5, so any suggestion is > appreciated. > > Thanks, > weiwei wang > From balay at mcs.anl.gov Mon Aug 10 09:12:31 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Aug 2009 09:12:31 -0500 (CDT) Subject: cannot find -lustdio with matlab on In-Reply-To: <4A800FEB.6060206@gmail.com> References: <4A800FEB.6060206@gmail.com> Message-ID: You don't need to install petsc with matlab for this. You can just use '-mat_view_matlab' - or PETSC_VIEWER_MATLAB in your code. Satish On Mon, 10 Aug 2009, Weiwei Wang wrote: > Dear All, > I want to check the determinant of some sparse matrix in petsc but I > didn't know how to do it using petsc directly so I try to save the sparse > matrix in Matlab format, then calculate the determinant in Matlab, so I > reinstalled petsc with the opinion "--with-matlab-dir=...", but the following > error appeared: > > /usr/bin/ld: cannot find -lustdio > collect2: ld returned 1 exit status > make[2]: [shared_linux] Error 1 (ignored) > > I don't know where I can find the lib "ustdio" or other ways to calculate the > determinant, the system of my box is debian 5, so any suggestion is > appreciated. > > Thanks, > weiwei wang > From thomas.witkowski at tu-dresden.de Mon Aug 10 10:14:19 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 10 Aug 2009 17:14:19 +0200 Subject: How to get information about the off-diagonal submatrix In-Reply-To: References: <4A802078.4040908@tu-dresden.de> Message-ID: <4A80394B.5050305@tu-dresden.de> Thanks, that works fine! In my test case (yes, just for debugging!) I've expected to have an empty off-diagonal matrix. But the matrix is not empty. But how is it possible, when all entries, which are inserted to the matrix via MatSetValues, have indices for row and col within the ownership range of the processors matrix? Thomas Satish Balay wrote: > alternatively - just access the private datastructe and print.. > > #include "../src/mat/impls/aij/mpi/mpiaij.h" > Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; > MatView(mpiaij->B,...); > > > Satish > > > On Mon, 10 Aug 2009, Matthew Knepley wrote: > > >> If this is just for debugging, the easiest way I think is to call >> MatGetSubmatrix() with >> a column IS that omits the local columns. >> >> Matt >> >> On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < >> thomas.witkowski at tu-dresden.de> wrote: >> >> >>> Hi, >>> >>> how is it possible to print just the off-diagonal submatrix of one process >>> of a parallel mpiaij matrix? >>> >>> Thomas >>> >>> >> >> >> > > > > From balay at mcs.anl.gov Mon Aug 10 10:29:05 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Aug 2009 10:29:05 -0500 (CDT) Subject: How to get information about the off-diagonal submatrix In-Reply-To: <4A80394B.5050305@tu-dresden.de> References: <4A802078.4040908@tu-dresden.de> <4A80394B.5050305@tu-dresden.de> Message-ID: You'll have to look at the code that does MatSetValues(). Perhaps you can run the code in debugger - and set a break point in MatSetValues_MPIAIJ - and see when MatSetValues_SeqAIJ_B_Private() is getting called.. Its also possible that some values are set on other procs and commuincated.. Note: I should have mentioned earlier: mpiaij->B is an internal datastructure - so the column indices don't directly correspond to the original matrix. [there is a cmap in there that stores this map] Satish On Mon, 10 Aug 2009, Thomas Witkowski wrote: > Thanks, that works fine! In my test case (yes, just for debugging!) I've > expected to have an empty off-diagonal matrix. But the matrix is not empty. > But how is it possible, when all entries, which are inserted to the matrix via > MatSetValues, have indices for row and col within the ownership range of the > processors matrix? > > Thomas > > Satish Balay wrote: > > alternatively - just access the private datastructe and print.. > > > > #include "../src/mat/impls/aij/mpi/mpiaij.h" > > Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; > > MatView(mpiaij->B,...); > > > > > > Satish > > > > > > On Mon, 10 Aug 2009, Matthew Knepley wrote: > > > > > > > If this is just for debugging, the easiest way I think is to call > > > MatGetSubmatrix() with > > > a column IS that omits the local columns. > > > > > > Matt > > > > > > On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < > > > thomas.witkowski at tu-dresden.de> wrote: > > > > > > > > > > Hi, > > > > > > > > how is it possible to print just the off-diagonal submatrix of one > > > > process > > > > of a parallel mpiaij matrix? > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > > > > From thomas.witkowski at tu-dresden.de Mon Aug 10 10:34:55 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 10 Aug 2009 17:34:55 +0200 Subject: How to get information about the off-diagonal submatrix In-Reply-To: References: <4A802078.4040908@tu-dresden.de> <4A80394B.5050305@tu-dresden.de> Message-ID: <4A803E1F.8070605@tu-dresden.de> Satish Balay wrote: > Its also possible that some values are set on other procs and > commuincated.. > That's the reason, I've forgotten to check it. Thanks for the hint! Is there any way to get information from petsc to check how many matrix entries had to be communicated during assembling? Thomas > Note: I should have mentioned earlier: mpiaij->B is an internal > datastructure - so the column indices don't directly correspond to the > original matrix. [there is a cmap in there that stores this map] > > Satish > > On Mon, 10 Aug 2009, Thomas Witkowski wrote: > > >> Thanks, that works fine! In my test case (yes, just for debugging!) I've >> expected to have an empty off-diagonal matrix. But the matrix is not empty. >> But how is it possible, when all entries, which are inserted to the matrix via >> MatSetValues, have indices for row and col within the ownership range of the >> processors matrix? >> >> Thomas >> >> Satish Balay wrote: >> >>> alternatively - just access the private datastructe and print.. >>> >>> #include "../src/mat/impls/aij/mpi/mpiaij.h" >>> Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; >>> MatView(mpiaij->B,...); >>> >>> >>> Satish >>> >>> >>> On Mon, 10 Aug 2009, Matthew Knepley wrote: >>> >>> >>> >>>> If this is just for debugging, the easiest way I think is to call >>>> MatGetSubmatrix() with >>>> a column IS that omits the local columns. >>>> >>>> Matt >>>> >>>> On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < >>>> thomas.witkowski at tu-dresden.de> wrote: >>>> >>>> >>>> >>>>> Hi, >>>>> >>>>> how is it possible to print just the off-diagonal submatrix of one >>>>> process >>>>> of a parallel mpiaij matrix? >>>>> >>>>> Thomas >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >> > > > > From balay at mcs.anl.gov Mon Aug 10 10:38:53 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 10 Aug 2009 10:38:53 -0500 (CDT) Subject: How to get information about the off-diagonal submatrix In-Reply-To: <4A803E1F.8070605@tu-dresden.de> References: <4A802078.4040908@tu-dresden.de> <4A80394B.5050305@tu-dresden.de> <4A803E1F.8070605@tu-dresden.de> Message-ID: On Mon, 10 Aug 2009, Thomas Witkowski wrote: > Satish Balay wrote: > > Its also possible that some values are set on other procs and > > commuincated.. > > > That's the reason, I've forgotten to check it. Thanks for the hint! Is there > any way to get information from petsc to check how many matrix entries had to > be communicated during assembling? You can run the code with '-info' - and look for 'stash' - that gives the message lenghts that got commuicated for matrix assembly. Also - you can use MatSetOption(MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE). This will cause MatSetValues() to ignore off-proc entries during assembly. After this - you can check if the assembled matrix is what you expect.. Satish > > Thomas > > > > > Note: I should have mentioned earlier: mpiaij->B is an internal > > datastructure - so the column indices don't directly correspond to the > > original matrix. [there is a cmap in there that stores this map] > > > > Satish > > > > On Mon, 10 Aug 2009, Thomas Witkowski wrote: > > > > > > > Thanks, that works fine! In my test case (yes, just for debugging!) I've > > > expected to have an empty off-diagonal matrix. But the matrix is not > > > empty. > > > But how is it possible, when all entries, which are inserted to the matrix > > > via > > > MatSetValues, have indices for row and col within the ownership range of > > > the > > > processors matrix? > > > > > > Thomas > > > > > > Satish Balay wrote: > > > > > > > alternatively - just access the private datastructe and print.. > > > > > > > > #include "../src/mat/impls/aij/mpi/mpiaij.h" > > > > Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; > > > > MatView(mpiaij->B,...); > > > > > > > > > > > > Satish > > > > > > > > > > > > On Mon, 10 Aug 2009, Matthew Knepley wrote: > > > > > > > > > > > > > If this is just for debugging, the easiest way I think is to call > > > > > MatGetSubmatrix() with > > > > > a column IS that omits the local columns. > > > > > > > > > > Matt > > > > > > > > > > On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < > > > > > thomas.witkowski at tu-dresden.de> wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > how is it possible to print just the off-diagonal submatrix of one > > > > > > process > > > > > > of a parallel mpiaij matrix? > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From thomas.witkowski at tu-dresden.de Mon Aug 10 10:46:09 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Mon, 10 Aug 2009 17:46:09 +0200 Subject: How to get information about the off-diagonal submatrix In-Reply-To: References: <4A802078.4040908@tu-dresden.de> <4A80394B.5050305@tu-dresden.de> <4A803E1F.8070605@tu-dresden.de> Message-ID: <4A8040C1.5090104@tu-dresden.de> Thank you for all the hints, this will help me to debug my application!!! Thomas Satish Balay wrote: > On Mon, 10 Aug 2009, Thomas Witkowski wrote: > > >> Satish Balay wrote: >> >>> Its also possible that some values are set on other procs and >>> commuincated.. >>> >>> >> That's the reason, I've forgotten to check it. Thanks for the hint! Is there >> any way to get information from petsc to check how many matrix entries had to >> be communicated during assembling? >> > > You can run the code with '-info' - and look for 'stash' - that gives > the message lenghts that got commuicated for matrix assembly. > > Also - you can use MatSetOption(MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE). > This will cause MatSetValues() to ignore off-proc entries during > assembly. After this - you can check if the assembled matrix is what > you expect.. > > Satish > > > >> Thomas >> >> >> >> >>> Note: I should have mentioned earlier: mpiaij->B is an internal >>> datastructure - so the column indices don't directly correspond to the >>> original matrix. [there is a cmap in there that stores this map] >>> >>> Satish >>> >>> On Mon, 10 Aug 2009, Thomas Witkowski wrote: >>> >>> >>> >>>> Thanks, that works fine! In my test case (yes, just for debugging!) I've >>>> expected to have an empty off-diagonal matrix. But the matrix is not >>>> empty. >>>> But how is it possible, when all entries, which are inserted to the matrix >>>> via >>>> MatSetValues, have indices for row and col within the ownership range of >>>> the >>>> processors matrix? >>>> >>>> Thomas >>>> >>>> Satish Balay wrote: >>>> >>>> >>>>> alternatively - just access the private datastructe and print.. >>>>> >>>>> #include "../src/mat/impls/aij/mpi/mpiaij.h" >>>>> Mat_MPIAIJ *mpiaij = (Mat_MPIAIJ*)mat->data; >>>>> MatView(mpiaij->B,...); >>>>> >>>>> >>>>> Satish >>>>> >>>>> >>>>> On Mon, 10 Aug 2009, Matthew Knepley wrote: >>>>> >>>>> >>>>> >>>>>> If this is just for debugging, the easiest way I think is to call >>>>>> MatGetSubmatrix() with >>>>>> a column IS that omits the local columns. >>>>>> >>>>>> Matt >>>>>> >>>>>> On Mon, Aug 10, 2009 at 8:28 AM, Thomas Witkowski < >>>>>> thomas.witkowski at tu-dresden.de> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> how is it possible to print just the off-diagonal submatrix of one >>>>>>> process >>>>>>> of a parallel mpiaij matrix? >>>>>>> >>>>>>> Thomas >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >> > > > > From mossaiby at yahoo.com Mon Aug 10 16:16:22 2009 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Mon, 10 Aug 2009 14:16:22 -0700 (PDT) Subject: Off topic - Some advice on solving Navier-Stokes with Finite Difference In-Reply-To: Message-ID: <655552.9097.qm@web52211.mail.re2.yahoo.com> Hi all, Sorry for this off-topic post. I am helping a master studnet which is working on solving Navier-Stokes equation with Finite Difference method. She is trying to eliminate spourious pressure modes from the solution. She needs to know some details that are not usually found in the papers but important when programming, e.g. boundary condition for pressure. If someone has expertise on this or know a *simple* FD code, I would be thankful to let me know. Best regards, Farshid Mossaiby From wangww2011 at gmail.com Mon Aug 10 22:10:50 2009 From: wangww2011 at gmail.com (Weiwei Wang) Date: Tue, 11 Aug 2009 11:10:50 +0800 Subject: cannot find -lustdio with matlab on In-Reply-To: References: <4A800FEB.6060206@gmail.com> Message-ID: <4A80E13A.1090900@gmail.com> Thanks a lot, it's ok using the code PetscViewer viewer; PetscViewerASCIIOpen(PETSC_COMM_WORLD, "K1.m", &viewer); PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); MatView(data->K1, viewer); but failed with the code MatView(data->K1, PETSC_VIEWER_MATLAB_WORLD); as the document say "Note: Currently can only save PETSc vectors to .mat files, not matrices" weiwei wang Hong Zhang wrote: > > weiwei, > > The easist way is run your petsc code with the option > '-mat_view_matlab', and dump it into a file. > Then use Matlab to load it for whatever you want to investigate. > See petsc user manual on "Using Matlab with PETSc". > > Hong > > On Mon, 10 Aug 2009, Weiwei Wang wrote: > >> Dear All, >> I want to check the determinant of some sparse matrix in petsc >> but I didn't know how to do it using petsc directly so I try to save >> the sparse matrix in Matlab format, then calculate the determinant in >> Matlab, so I reinstalled petsc with the opinion >> "--with-matlab-dir=...", but the following error appeared: >> >> /usr/bin/ld: cannot find -lustdio >> collect2: ld returned 1 exit status >> make[2]: [shared_linux] Error 1 (ignored) >> >> I don't know where I can find the lib "ustdio" or other ways to >> calculate the determinant, the system of my box is debian 5, so any >> suggestion is appreciated. >> >> Thanks, >> weiwei wang >> > From bsmith at mcs.anl.gov Mon Aug 10 22:44:55 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 10 Aug 2009 22:44:55 -0500 Subject: cannot find -lustdio with matlab on In-Reply-To: <4A80E13A.1090900@gmail.com> References: <4A800FEB.6060206@gmail.com> <4A80E13A.1090900@gmail.com> Message-ID: <137E0B26-A01A-4086-B0CA-921AFF126233@mcs.anl.gov> You should really use the PetscViewerBinaryOpen() then use bin/ matlab/PetscBinaryRead() to read the matrices and vectors into Matlab. Barry On Aug 10, 2009, at 10:10 PM, Weiwei Wang wrote: > Thanks a lot, it's ok using the code > PetscViewer viewer; > PetscViewerASCIIOpen(PETSC_COMM_WORLD, "K1.m", &viewer); > PetscViewerSetFormat(viewer, PETSC_VIEWER_ASCII_MATLAB); > MatView(data->K1, viewer); > but failed with the code > MatView(data->K1, PETSC_VIEWER_MATLAB_WORLD); > as the document say "Note: Currently can only save PETSc vectors > to .mat files, not matrices" > > weiwei wang > > Hong Zhang wrote: >> >> weiwei, >> >> The easist way is run your petsc code with the option >> '-mat_view_matlab', and dump it into a file. >> Then use Matlab to load it for whatever you want to investigate. >> See petsc user manual on "Using Matlab with PETSc". >> >> Hong >> >> On Mon, 10 Aug 2009, Weiwei Wang wrote: >> >>> Dear All, >>> I want to check the determinant of some sparse matrix in petsc >>> but I didn't know how to do it using petsc directly so I try to >>> save the sparse matrix in Matlab format, then calculate the >>> determinant in Matlab, so I reinstalled petsc with the opinion "-- >>> with-matlab-dir=...", but the following error appeared: >>> >>> /usr/bin/ld: cannot find -lustdio >>> collect2: ld returned 1 exit status >>> make[2]: [shared_linux] Error 1 (ignored) >>> >>> I don't know where I can find the lib "ustdio" or other ways to >>> calculate the determinant, the system of my box is debian 5, so >>> any suggestion is appreciated. >>> >>> Thanks, >>> weiwei wang >>> >> > From wangww2011 at gmail.com Tue Aug 11 02:13:19 2009 From: wangww2011 at gmail.com (Weiwei Wang) Date: Tue, 11 Aug 2009 15:13:19 +0800 Subject: cannot find -lustdio with matlab on In-Reply-To: <137E0B26-A01A-4086-B0CA-921AFF126233@mcs.anl.gov> References: <4A800FEB.6060206@gmail.com> <4A80E13A.1090900@gmail.com> <137E0B26-A01A-4086-B0CA-921AFF126233@mcs.anl.gov> Message-ID: <4A811A0F.7010400@gmail.com> That's a good point, Thank you very much. Barry Smith wrote: > > You should really use the PetscViewerBinaryOpen() then use > bin/matlab/PetscBinaryRead() to read the matrices and vectors into > Matlab. > > Barry > > On Aug 10, 2009, at 10:10 PM, Weiwei Wang wrote: > From joegi.geo at yahoo.com Tue Aug 11 20:48:42 2009 From: joegi.geo at yahoo.com (Joel Guerrero) Date: Tue, 11 Aug 2009 18:48:42 -0700 (PDT) Subject: porting petsc to a nec sx9 computer running under unix super ux Message-ID: <100492.19832.qm@web34405.mail.mud.yahoo.com> Hi, I wonder if somebody have experience in porting petsc to a nec sx9 computer? Cheers, joel From balay at mcs.anl.gov Tue Aug 11 21:29:26 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 11 Aug 2009 21:29:26 -0500 (CDT) Subject: porting petsc to a nec sx9 computer running under unix super ux In-Reply-To: <100492.19832.qm@web34405.mail.mud.yahoo.com> References: <100492.19832.qm@web34405.mail.mud.yahoo.com> Message-ID: we had an old version - perhaps 2.2.x working on sx6. It has standard compilers with mpi and blas - so I'm guessing current petsc should work on sx9. If you encounter problems building on sx9 - then send us the error logs at petsc-maint at mcs.anl.gov Satish On Tue, 11 Aug 2009, Joel Guerrero wrote: > Hi, > > I wonder if somebody have experience in porting petsc to a nec sx9 computer? > > > Cheers, > > joel > > > > > From joegi.geo at yahoo.com Tue Aug 11 21:46:21 2009 From: joegi.geo at yahoo.com (Joel Guerrero) Date: Tue, 11 Aug 2009 19:46:21 -0700 (PDT) Subject: porting petsc to a nec sx9 computer running under unix super ux In-Reply-To: References: <100492.19832.qm@web34405.mail.mud.yahoo.com> Message-ID: <78389.75985.qm@web34406.mail.mud.yahoo.com> Hi Satish, Well basically I am having problems with the cross compilation, are there any instructions available about how to cross compile petsc on these platforms?. I am using sx9 compilers (sxcc, sxc++, sxmpicc and so on). Thanks for your help, Joel ----- Original Message ---- From: Satish Balay To: PETSc users list Sent: Wednesday, August 12, 2009 4:29:26 AM Subject: Re: porting petsc to a nec sx9 computer running under unix super ux we had an old version - perhaps 2.2.x working on sx6. It has standard compilers with mpi and blas - so I'm guessing current petsc should work on sx9. If you encounter problems building on sx9 - then send us the error logs at petsc-maint at mcs.anl.gov Satish On Tue, 11 Aug 2009, Joel Guerrero wrote: > Hi, > > I wonder if somebody have experience in porting petsc to a nec sx9 computer? > > > Cheers, > > joel > > > > > From bsmith at mcs.anl.gov Tue Aug 11 21:49:48 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 11 Aug 2009 21:49:48 -0500 Subject: porting petsc to a nec sx9 computer running under unix super ux In-Reply-To: <78389.75985.qm@web34406.mail.mud.yahoo.com> References: <100492.19832.qm@web34405.mail.mud.yahoo.com> <78389.75985.qm@web34406.mail.mud.yahoo.com> Message-ID: <76004F3C-83FF-41AD-8A57-7793B50A9B15@mcs.anl.gov> Run ./configure with the additional option --with-batch and then follow the directions Barry On Aug 11, 2009, at 9:46 PM, Joel Guerrero wrote: > Hi Satish, > > Well basically I am having problems with the cross compilation, are > there any instructions available about how to cross compile petsc on > these platforms?. I am using sx9 compilers (sxcc, sxc++, sxmpicc > and so on). > > > Thanks for your help, > > Joel > > > > > ----- Original Message ---- > From: Satish Balay > To: PETSc users list > Sent: Wednesday, August 12, 2009 4:29:26 AM > Subject: Re: porting petsc to a nec sx9 computer running under unix > super ux > > we had an old version - perhaps 2.2.x working on sx6. It has standard > compilers with mpi and blas - so I'm guessing current petsc should > work on sx9. > > If you encounter problems building on sx9 - then send us the error > logs at petsc-maint at mcs.anl.gov > > Satish > > On Tue, 11 Aug 2009, Joel Guerrero wrote: > >> Hi, >> >> I wonder if somebody have experience in porting petsc to a nec sx9 >> computer? >> >> >> Cheers, >> >> joel >> >> >> >> >> > > > From balay at mcs.anl.gov Tue Aug 11 21:54:40 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 11 Aug 2009 21:54:40 -0500 (CDT) Subject: porting petsc to a nec sx9 computer running under unix super ux In-Reply-To: <78389.75985.qm@web34406.mail.mud.yahoo.com> References: <100492.19832.qm@web34405.mail.mud.yahoo.com> <78389.75985.qm@web34406.mail.mud.yahoo.com> Message-ID: Try using --with-batch=1 and follow the sugested steps. Looking at the sx6 config file - looks like the compilers used were cc,f90,cxx - so perhaps the env was slightly different. [don't remember if it was cross compile or not..] One potential issue is: configure might not be able to auto-detect c/fortran compiler libraries [needed for mixing c & fortran codes]. If this is an issue - sugest using someting like: --with-clib-autodetect=0 --with-fortranlib-autodetect=0 --with-cxxlib-autodetect=0 LIBS="-lf90sx -lf90sxe -lv90sxe -li90sx -lu90sx -lm90sx" [Might have to modify LIBS argument to the correct c/fortran compiler library list] If you encounter problems - send us configure.log at petsc-maint at mcs.anl.gov Satish On Tue, 11 Aug 2009, Joel Guerrero wrote: > Hi Satish, > > Well basically I am having problems with the cross compilation, are there any instructions available about how to cross compile petsc on these platforms?. I am using sx9 compilers (sxcc, sxc++, sxmpicc and so on). > > > Thanks for your help, > > Joel > > > > > ----- Original Message ---- > From: Satish Balay > To: PETSc users list > Sent: Wednesday, August 12, 2009 4:29:26 AM > Subject: Re: porting petsc to a nec sx9 computer running under unix super ux > > we had an old version - perhaps 2.2.x working on sx6. It has standard > compilers with mpi and blas - so I'm guessing current petsc should > work on sx9. > > If you encounter problems building on sx9 - then send us the error > logs at petsc-maint at mcs.anl.gov > > Satish > > On Tue, 11 Aug 2009, Joel Guerrero wrote: > > > Hi, > > > > I wonder if somebody have experience in porting petsc to a nec sx9 computer? > > > > > > Cheers, > > > > joel > > > > > > > > > > > > > > From lua.byhh at gmail.com Tue Aug 11 23:37:32 2009 From: lua.byhh at gmail.com (Shengyong) Date: Wed, 12 Aug 2009 12:37:32 +0800 Subject: Off topic - Some advice on solving Navier-Stokes with Finite Difference In-Reply-To: <655552.9097.qm@web52211.mail.re2.yahoo.com> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> Message-ID: Hi, Farshid Maybe she should use the staggered grid method which is very simple to implement. On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby wrote: > Hi all, > > Sorry for this off-topic post. > > I am helping a master studnet which is working on solving Navier-Stokes > equation with Finite Difference method. She is trying to eliminate spourious > pressure modes from the solution. She needs to know some details that are > not usually found in the papers but important when programming, e.g. > boundary condition for pressure. If someone has expertise on this or know a > *simple* FD code, I would be thankful to let me know. > > Best regards, > Farshid Mossaiby > > > > -- Pang Shengyong Solidification Simulation Lab, State Key Lab of Mould & Die Technology, Huazhong Univ. of Sci. & Tech. China -------------- next part -------------- An HTML attachment was scrubbed... URL: From mossaiby at yahoo.com Wed Aug 12 01:06:52 2009 From: mossaiby at yahoo.com (Farshid Mossaiby) Date: Tue, 11 Aug 2009 23:06:52 -0700 (PDT) Subject: Off topic - Some advice on solving Navier-Stokes with Finite Difference In-Reply-To: Message-ID: <640746.98196.qm@web52206.mail.re2.yahoo.com> Hi Shengyong, Thanks for the suggestion. Of course we know about the staggered grid method, but it is a research work and she is working to improve the method without using the staggered grid. Any suggestion would be welcome (maybe via private email). Again sorry for this off-topic post. Best regards, Farshid Mossaiby --- On Wed, 8/12/09, Shengyong wrote: > From: Shengyong > Subject: Re: Off topic - Some advice on solving Navier-Stokes with Finite Difference > To: "PETSc users list" > Date: Wednesday, August 12, 2009, 9:07 AM > Hi, Farshid > > Maybe she should use the staggered grid method which is > very simple to implement. > > On Tue, Aug 11, 2009 at 5:16 AM, > Farshid Mossaiby > wrote: > > Hi all, > > > > Sorry for this off-topic post. > > > > I am helping a master studnet which is working on solving > Navier-Stokes equation with Finite Difference method. She is > trying to eliminate spourious pressure modes from the > solution. She needs to know some details that are not > usually found in the papers but important when programming, > e.g. boundary condition for pressure. If someone has > expertise on this or know a *simple* FD code, I would be > thankful to let me know. > > > > > Best regards, > > Farshid Mossaiby > > > > > > > > > > > -- > Pang Shengyong > Solidification ?Simulation Lab, > State Key Lab of ?Mould & Die Technology, > Huazhong Univ. of Sci. & Tech. China > > From fredrik.bengzon at math.umu.se Wed Aug 12 08:23:21 2009 From: fredrik.bengzon at math.umu.se (Fredrik Bengzon) Date: Wed, 12 Aug 2009 15:23:21 +0200 Subject: methods on adjacency matrix Message-ID: <4A82C249.7000409@math.umu.se> Hi, I'm trying to minimize my code managing a finite element mesh and thought of using the adjacency matrix type. Is there a list of methods working on this matrix type anywhere in the documentation? I want to store the node-to-element information in such a matrix and then use the MatGetSubMatrices and MatGetRow methods. Is this possible? Thanks. /Fredrik Bengzon From knepley at gmail.com Wed Aug 12 08:47:42 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Aug 2009 08:47:42 -0500 Subject: methods on adjacency matrix In-Reply-To: <4A82C249.7000409@math.umu.se> References: <4A82C249.7000409@math.umu.se> Message-ID: All the matrices have the same interface. Adj is just a different storage format. Matt On Wed, Aug 12, 2009 at 8:23 AM, Fredrik Bengzon < fredrik.bengzon at math.umu.se> wrote: > Hi, > I'm trying to minimize my code managing a finite element mesh and thought > of using the adjacency matrix type. Is there a list of methods working on > this matrix type anywhere in the documentation? I want to store the > node-to-element information in such a matrix and then use the > MatGetSubMatrices and MatGetRow methods. Is this possible? Thanks. > /Fredrik Bengzon > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Aug 12 09:14:58 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 12 Aug 2009 09:14:58 -0500 (CDT) Subject: porting petsc to a nec sx9 computer running under unix super ux In-Reply-To: <17650.39337.qm@web34403.mail.mud.yahoo.com> References: <100492.19832.qm@web34405.mail.mud.yahoo.com> <78389.75985.qm@web34406.mail.mud.yahoo.com> <17650.39337.qm@web34403.mail.mud.yahoo.com> Message-ID: Please send configure.log to petsc-maint - not the mailing list. Why use petsc-2.3.2-p3 - instead of the latest petsc-3.0.0? On Wed, 12 Aug 2009, Joel Guerrero wrote: > Hi Satish, > > I am trying to compile petsc using the following configuration, > > ./config/configure.py --PETSC_ARCH=sx9-nec-superux --with-debugging=0 --with-fortran=0 --with-matlab=0 --with-mpi=0 --with-shared=1 --with-dynamic=1 --with-CC=sxcc --CFLAGS="-Chopt -verbose -pi" --with-batch=1 > the option is --with-cc [or CC] - not --with-CC. Satish > I am attaching the configure.log. Btw, I am using the system optimize lapack/blas libraries. > > Hope you can help me, > > Joel > > > > > > > > ----- Original Message ---- > From: Satish Balay > To: PETSc users list > Sent: Wednesday, August 12, 2009 4:54:40 AM > Subject: Re: porting petsc to a nec sx9 computer running under unix super ux > > Try using --with-batch=1 and follow the sugested steps. > > Looking at the sx6 config file - looks like the compilers used were > cc,f90,cxx - so perhaps the env was slightly different. [don't > remember if it was cross compile or not..] > > One potential issue is: configure might not be able to auto-detect > c/fortran compiler libraries [needed for mixing c & fortran codes]. If > this is an issue - sugest using someting like: > > --with-clib-autodetect=0 > --with-fortranlib-autodetect=0 > --with-cxxlib-autodetect=0 > LIBS="-lf90sx -lf90sxe -lv90sxe -li90sx -lu90sx -lm90sx" > > [Might have to modify LIBS argument to the correct c/fortran compiler library list] > > If you encounter problems - send us configure.log at petsc-maint at mcs.anl.gov > > Satish > > > On Tue, 11 Aug 2009, Joel Guerrero wrote: > > > Hi Satish, > > > > Well basically I am having problems with the cross compilation, are there any instructions available about how to cross compile petsc on these platforms?. I am using sx9 compilers (sxcc, sxc++, sxmpicc and so on). > > > > > > Thanks for your help, > > > > Joel > > > > > > > > > > ----- Original Message ---- > > From: Satish Balay > > To: PETSc users list > > Sent: Wednesday, August 12, 2009 4:29:26 AM > > Subject: Re: porting petsc to a nec sx9 computer running under unix super ux > > > > we had an old version - perhaps 2.2.x working on sx6. It has standard > > compilers with mpi and blas - so I'm guessing current petsc should > > work on sx9. > > > > If you encounter problems building on sx9 - then send us the error > > logs at petsc-maint at mcs.anl.gov > > > > Satish > > > > On Tue, 11 Aug 2009, Joel Guerrero wrote: > > > > > Hi, > > > > > > I wonder if somebody have experience in porting petsc to a nec sx9 computer? > > > > > > > > > Cheers, > > > > > > joel > > > > > > > > > > > > > > > > > > > > > > > > > > From stephen.wornom at sophia.inria.fr Wed Aug 12 09:30:53 2009 From: stephen.wornom at sophia.inria.fr (Stephen Wornom) Date: Wed, 12 Aug 2009 16:30:53 +0200 Subject: Off topic - Some advice on solving Navier-Stokes with Finite Difference In-Reply-To: References: <655552.9097.qm@web52211.mail.re2.yahoo.com> Message-ID: <4A82D21D.3070409@sophia.inria.fr> Shengyong wrote: > Hi, Farshid > > Maybe she should use the staggered grid method which is very simple to > implement. Does it remain simple for curvilinear meshes? Stephen > > On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby > wrote: > > Hi all, > > Sorry for this off-topic post. > > I am helping a master studnet which is working on solving > Navier-Stokes equation with Finite Difference method. She is > trying to eliminate spourious pressure modes from the solution. > She needs to know some details that are not usually found in the > papers but important when programming, e.g. boundary condition for > pressure. If someone has expertise on this or know a *simple* FD > code, I would be thankful to let me know. > > Best regards, > Farshid Mossaiby > > > > > > > -- > Pang Shengyong > Solidification Simulation Lab, > State Key Lab of Mould & Die Technology, > Huazhong Univ. of Sci. & Tech. China From william.perkins at pnl.gov Wed Aug 12 10:19:07 2009 From: william.perkins at pnl.gov (William A. Perkins) Date: Wed, 12 Aug 2009 08:19:07 -0700 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: <4A82D21D.3070409@sophia.inria.fr> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> Message-ID: <19074.56683.949429.316119@bearflag.pnl.gov> Stephen, There are two ways that I know of to deal with pressure checker boarding: staggered grids or some form of Rhie-Chow interpolation. IMO, these are simple only for uniform, Cartesian grids. For grids that are curvilinear, unstructured, non-uniform, and/or non-orthogonal, things get real complicated. There may be other methods, but something is required. Regarding boundary conditions, I would suggest this text book: H. K. Versteeg and W. Malalasekera. An Introduction to Computational Fluid Dynamics, the Finite Volume Method. 2nd edition. Prentice-Hall. 2007 While this book uses the finite volume method, the explanation of boundary conditions and staggered grids is very good and relatively easy to interpret for finite difference. I would also recommend Joel H. Ferziger and Milovan Peric. Computational Methods for Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. This is a little more general with regard to method discussing finite difference and finite volume, but still settling on finite volume. My $0.02: I question the use of finite difference. For Navier-Stokes, the use of finite volume is much more prevalent in commercial and research codes. If your student follows Versteeg and Malalasekera a simple, working, staggered grid FV code could be built in a very short time. If something more complicated is needed, it's probably explained in Ferziger and Peric. Also My $0.02: Unless the point of your student's work is to experience building her own code, why not download something like OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use it? I expect the effort to learn something like OpenFOAM for a simple application will be much less than writing a new code. Hope this helps. Bill >>>>> "Stephen" == Stephen Wornom writes: Stephen> Shengyong wrote: >> Hi, Farshid >> >> Maybe she should use the staggered grid method which is very simple to >> implement. Stephen> Does it remain simple for curvilinear meshes? Stephen> Stephen >> >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby > > wrote: >> >> Hi all, >> >> Sorry for this off-topic post. >> >> I am helping a master studnet which is working on solving >> Navier-Stokes equation with Finite Difference method. She is >> trying to eliminate spourious pressure modes from the solution. >> She needs to know some details that are not usually found in the >> papers but important when programming, e.g. boundary condition for >> pressure. If someone has expertise on this or know a *simple* FD >> code, I would be thankful to let me know. >> >> Best regards, >> Farshid Mossaiby >> >> >> >> >> >> >> -- >> Pang Shengyong >> Solidification Simulation Lab, >> State Key Lab of Mould & Die Technology, >> Huazhong Univ. of Sci. & Tech. China -- Bill Perkins Research Engineer Hydrology Group Pacific Northwest National Laboratory 902 Battelle Boulevard P.O. Box 999, MSIN K9-36 Richland, WA 99352 USA Tel: 509-372-6131 Fax: 509-372-6089 william.perkins at pnl.gov www.pnl.gov From knepley at gmail.com Wed Aug 12 10:29:19 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Aug 2009 10:29:19 -0500 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: <19074.56683.949429.316119@bearflag.pnl.gov> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> Message-ID: 1) You should really handle this by creating the constant vector on the pressure space and using MatNullSpaceCreate() 2) You can also easily handle this by fixing the pressure at one point Matt On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins < william.perkins at pnl.gov> wrote: > > Stephen, > > There are two ways that I know of to deal with pressure checker > boarding: staggered grids or some form of Rhie-Chow interpolation. > IMO, these are simple only for uniform, Cartesian grids. For grids > that are curvilinear, unstructured, non-uniform, and/or > non-orthogonal, things get real complicated. There may be other > methods, but something is required. > > Regarding boundary conditions, I would suggest this text book: > > H. K. Versteeg and W. Malalasekera. An Introduction to > Computational Fluid Dynamics, the Finite Volume Method. 2nd > edition. Prentice-Hall. 2007 > > While this book uses the finite volume method, the explanation of > boundary conditions and staggered grids is very good and relatively > easy to interpret for finite difference. I would also recommend > > Joel H. Ferziger and Milovan Peric. Computational Methods for > Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. > > This is a little more general with regard to method discussing finite > difference and finite volume, but still settling on finite volume. > > My $0.02: I question the use of finite difference. For Navier-Stokes, > the use of finite volume is much more prevalent in commercial and > research codes. If your student follows Versteeg and Malalasekera a > simple, working, staggered grid FV code could be built in a very short > time. If something more complicated is needed, it's probably > explained in Ferziger and Peric. > > Also My $0.02: Unless the point of your student's work is to > experience building her own code, why not download something like > OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use it? I > expect the effort to learn something like OpenFOAM for a simple > application will be much less than writing a new code. > > Hope this helps. > > Bill > > >>>>> "Stephen" == Stephen Wornom writes: > > Stephen> Shengyong wrote: > >> Hi, Farshid > >> > >> Maybe she should use the staggered grid method which is very simple > to > >> implement. > Stephen> Does it remain simple for curvilinear meshes? > Stephen> Stephen > >> > >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby < > mossaiby at yahoo.com > >> > wrote: > >> > >> Hi all, > >> > >> Sorry for this off-topic post. > >> > >> I am helping a master studnet which is working on solving > >> Navier-Stokes equation with Finite Difference method. She is > >> trying to eliminate spourious pressure modes from the solution. > >> She needs to know some details that are not usually found in the > >> papers but important when programming, e.g. boundary condition for > >> pressure. If someone has expertise on this or know a *simple* FD > >> code, I would be thankful to let me know. > >> > >> Best regards, > >> Farshid Mossaiby > >> > >> > >> > >> > >> > >> > >> -- > >> Pang Shengyong > >> Solidification Simulation Lab, > >> State Key Lab of Mould & Die Technology, > >> Huazhong Univ. of Sci. & Tech. China > > > -- > Bill Perkins > Research Engineer > Hydrology Group > > Pacific Northwest National Laboratory > 902 Battelle Boulevard > P.O. Box 999, MSIN K9-36 > Richland, WA 99352 USA > Tel: 509-372-6131 > Fax: 509-372-6089 > william.perkins at pnl.gov > www.pnl.gov > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.wornom at sophia.inria.fr Wed Aug 12 10:39:20 2009 From: stephen.wornom at sophia.inria.fr (Stephen Wornom) Date: Wed, 12 Aug 2009 17:39:20 +0200 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> Message-ID: <4A82E228.90400@sophia.inria.fr> Matthew Knepley wrote: > 1) You should really handle this by creating the constant vector on > the pressure > space and using MatNullSpaceCreate() > > 2) You can also easily handle this by fixing the pressure at one point At what indices or location does one fix the pressure? What value is it set. Usually pressure is part of the solution. It would seem to introduce an inconsistency. I would like to understand how to do it. Stephen > > Matt > > On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins > > wrote: > > > Stephen, > > There are two ways that I know of to deal with pressure checker > boarding: staggered grids or some form of Rhie-Chow interpolation. > IMO, these are simple only for uniform, Cartesian grids. For grids > that are curvilinear, unstructured, non-uniform, and/or > non-orthogonal, things get real complicated. There may be other > methods, but something is required. > > Regarding boundary conditions, I would suggest this text book: > > H. K. Versteeg and W. Malalasekera. An Introduction to > Computational Fluid Dynamics, the Finite Volume Method. 2nd > edition. Prentice-Hall. 2007 > > While this book uses the finite volume method, the explanation of > boundary conditions and staggered grids is very good and relatively > easy to interpret for finite difference. I would also recommend > > Joel H. Ferziger and Milovan Peric. Computational Methods for > Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. > > This is a little more general with regard to method discussing finite > difference and finite volume, but still settling on finite volume. > > My $0.02: I question the use of finite difference. For Navier-Stokes, > the use of finite volume is much more prevalent in commercial and > research codes. If your student follows Versteeg and Malalasekera a > simple, working, staggered grid FV code could be built in a very short > time. If something more complicated is needed, it's probably > explained in Ferziger and Peric. > > Also My $0.02: Unless the point of your student's work is to > experience building her own code, why not download something like > OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use it? I > expect the effort to learn something like OpenFOAM for a simple > application will be much less than writing a new code. > > Hope this helps. > > Bill > > >>>>> "Stephen" == Stephen Wornom > writes: > > Stephen> Shengyong wrote: > >> Hi, Farshid > >> > >> Maybe she should use the staggered grid method which is very > simple to > >> implement. > Stephen> Does it remain simple for curvilinear meshes? > Stephen> Stephen > >> > >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby > > >> >> wrote: > >> > >> Hi all, > >> > >> Sorry for this off-topic post. > >> > >> I am helping a master studnet which is working on solving > >> Navier-Stokes equation with Finite Difference method. She is > >> trying to eliminate spourious pressure modes from the solution. > >> She needs to know some details that are not usually found in the > >> papers but important when programming, e.g. boundary > condition for > >> pressure. If someone has expertise on this or know a *simple* FD > >> code, I would be thankful to let me know. > >> > >> Best regards, > >> Farshid Mossaiby > >> > >> > >> > >> > >> > >> > >> -- > >> Pang Shengyong > >> Solidification Simulation Lab, > >> State Key Lab of Mould & Die Technology, > >> Huazhong Univ. of Sci. & Tech. China > > > -- > Bill Perkins > Research Engineer > Hydrology Group > > Pacific Northwest National Laboratory > 902 Battelle Boulevard > P.O. Box 999, MSIN K9-36 > Richland, WA 99352 USA > Tel: 509-372-6131 > Fax: 509-372-6089 > william.perkins at pnl.gov > www.pnl.gov > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From knepley at gmail.com Wed Aug 12 11:00:10 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Aug 2009 11:00:10 -0500 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: <4A82E228.90400@sophia.inria.fr> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> <4A82E228.90400@sophia.inria.fr> Message-ID: On Wed, Aug 12, 2009 at 10:39 AM, Stephen Wornom < stephen.wornom at sophia.inria.fr> wrote: > Matthew Knepley wrote: > >> 1) You should really handle this by creating the constant vector on the >> pressure >> space and using MatNullSpaceCreate() >> >> 2) You can also easily handle this by fixing the pressure at one point >> > At what indices or location does one fix the pressure? What value is it > set. Usually pressure is part of the solution. It would seem to introduce an > inconsistency. > I would like to understand how to do it. It does not matter where you fix it or to what value. The absolute value of the pressure is not physically relevant, only pressure differences (which is why only grad p appears). Fixing the pressure at any point to any value just sets the scale. There is no inconsistency. This is in many many elementary fluid mechanics books. To do this mechanically. Change one row in the operator div (from the div u = 0 equation) to the identity. Matt > > Stephen > > >> Matt >> >> >> On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins < >> william.perkins at pnl.gov > wrote: >> >> >> Stephen, >> >> There are two ways that I know of to deal with pressure checker >> boarding: staggered grids or some form of Rhie-Chow interpolation. >> IMO, these are simple only for uniform, Cartesian grids. For grids >> that are curvilinear, unstructured, non-uniform, and/or >> non-orthogonal, things get real complicated. There may be other >> methods, but something is required. >> >> Regarding boundary conditions, I would suggest this text book: >> >> H. K. Versteeg and W. Malalasekera. An Introduction to >> Computational Fluid Dynamics, the Finite Volume Method. 2nd >> edition. Prentice-Hall. 2007 >> >> While this book uses the finite volume method, the explanation of >> boundary conditions and staggered grids is very good and relatively >> easy to interpret for finite difference. I would also recommend >> >> Joel H. Ferziger and Milovan Peric. Computational Methods for >> Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. >> >> This is a little more general with regard to method discussing finite >> difference and finite volume, but still settling on finite volume. >> >> My $0.02: I question the use of finite difference. For Navier-Stokes, >> the use of finite volume is much more prevalent in commercial and >> research codes. If your student follows Versteeg and Malalasekera a >> simple, working, staggered grid FV code could be built in a very short >> time. If something more complicated is needed, it's probably >> explained in Ferziger and Peric. >> >> Also My $0.02: Unless the point of your student's work is to >> experience building her own code, why not download something like >> OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use it? I >> expect the effort to learn something like OpenFOAM for a simple >> application will be much less than writing a new code. >> >> Hope this helps. >> >> Bill >> >> >>>>> "Stephen" == Stephen Wornom > > writes: >> >> Stephen> Shengyong wrote: >> >> Hi, Farshid >> >> >> >> Maybe she should use the staggered grid method which is very >> simple to >> >> implement. >> Stephen> Does it remain simple for curvilinear meshes? >> Stephen> Stephen >> >> >> >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby >> >> >> >> wrote: >> >> >> >> Hi all, >> >> >> >> Sorry for this off-topic post. >> >> >> >> I am helping a master studnet which is working on solving >> >> Navier-Stokes equation with Finite Difference method. She is >> >> trying to eliminate spourious pressure modes from the solution. >> >> She needs to know some details that are not usually found in the >> >> papers but important when programming, e.g. boundary >> condition for >> >> pressure. If someone has expertise on this or know a *simple* FD >> >> code, I would be thankful to let me know. >> >> >> >> Best regards, >> >> Farshid Mossaiby >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Pang Shengyong >> >> Solidification Simulation Lab, >> >> State Key Lab of Mould & Die Technology, >> >> Huazhong Univ. of Sci. & Tech. China >> >> >> -- >> Bill Perkins >> Research Engineer >> Hydrology Group >> >> Pacific Northwest National Laboratory >> 902 Battelle Boulevard >> P.O. Box 999, MSIN K9-36 >> Richland, WA 99352 USA >> Tel: 509-372-6131 >> Fax: 509-372-6089 >> william.perkins at pnl.gov >> www.pnl.gov >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 12 11:34:22 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 12 Aug 2009 11:34:22 -0500 Subject: methods on adjacency matrix In-Reply-To: References: <4A82C249.7000409@math.umu.se> Message-ID: <8989ADAF-28B8-4A39-951F-8CDE020A677C@mcs.anl.gov> On Aug 12, 2009, at 8:47 AM, Matthew Knepley wrote: > All the matrices have the same interface. Adj is just a different > storage format. Yes, but very few methods are implemented for that type. It is there to support calls to partitioning packages. The function table is in src/mat/impls/adj/mpi/mpiadj.c it has only MatGetRow_MPIAdj, MatRestoreRow_MPIAdj, MatEqual_MPIAdj, MatSetOption_MPIAdj, MatGetRowIJ_MPIAdj, MatRestoreRowIJ_MPIAdj, MatDestroy_MPIAdj, MatView_MPIAdj, MatConvertFrom_MPIAdj, You can use AIJ matrix type instead. Barry > > Matt > > On Wed, Aug 12, 2009 at 8:23 AM, Fredrik Bengzon > wrote: > Hi, > I'm trying to minimize my code managing a finite element mesh and > thought of using the adjacency matrix type. Is there a list of > methods working on this matrix type anywhere in the documentation? I > want to store the node-to-element information in such a matrix and > then use the MatGetSubMatrices and MatGetRow methods. Is this > possible? Thanks. > /Fredrik Bengzon > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From alianjun at us.ibm.com Wed Aug 12 13:01:58 2009 From: alianjun at us.ibm.com (Lianjun An) Date: Wed, 12 Aug 2009 14:01:58 -0400 Subject: AO scalability issue Message-ID: We currently use Petsc routine "AOCreateBasic" to create application order --- a mapping from used id to petsc id. Then we setup a connection graph for our mesh and assemble vector and matrix based on the Petsc order. When total number of elements in our mesh is about couple million and 64 computer nodes are used, AO is created in a reasonable time. But when we have 700 million elements in the mesh and use more then 256 computer nodes, the time to create AO grows significantly (9000 secs). It seems that each process has whole permutation table about the mapping and quite expensive in time and buffer size. a) Memory requirement is quite high (AO creation failed for 1.3 billion elements on 1024 nodes due to not enough memory). b) If user id numbering on each processor is partially continuous, then time for creation AO is reduced. c) In my understanding, AOApplicationToPetsc and AOPetscToApplication routines don't use communication since AO has whole mapping on each processor. In our application, we might not need the whole mapping in each processor. We only need to ids of vertices that reside in the processor (limited ghost vertices). We wonder whether there is an efficient way to create AO and to cut its creation time and buffer size. Let us know your thought on that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Aug 12 13:19:29 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 12 Aug 2009 13:19:29 -0500 Subject: AO scalability issue In-Reply-To: References: Message-ID: On Wed, Aug 12, 2009 at 1:01 PM, Lianjun An wrote: > We currently use Petsc routine "AOCreateBasic" to create application order > --- a mapping from used id to petsc id. Then we setup a connection graph for > our mesh and assemble vector and matrix based on the Petsc order. When total > number of elements in our mesh is about couple million and 64 computer nodes > are used, AO is created in a reasonable time. But when we have 700 million > elements in the mesh and use more then 256 computer nodes, the time to > create AO grows significantly (9000 secs). > > It seems that each process has whole permutation table about the mapping > and quite expensive in time and buffer size. > a) Memory requirement is quite high (AO creation failed for 1.3 billion > elements on 1024 nodes due to not enough memory). > b) If user id numbering on each processor is partially continuous, then > time for creation AO is reduced. > c) In my understanding, AOApplicationToPetsc and AOPetscToApplication > routines don't use communication since AO has whole mapping on each > processor. > > In our application, we might not need the whole mapping in each processor. > We only need to ids of vertices that reside in the processor (limited ghost > vertices). We wonder whether there is an efficient way to create AO and to > cut its creation time and buffer size. Let us know your thought on that. > The AO was not designed to be scalable (unfortunately). Any scalable version would segment the renumbering, so that each process was responsible for a range of entries. You would first communicate the indices to correct process (using a VecScatter), renumber, and then communicate the result back to the original process. It does not sound all that hard to me (using contiguous indices), however we are unlikely to implement it right now. If you would like to try yourself, we would be willing to help. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From aron.ahmadia at kaust.edu.sa Wed Aug 12 13:46:55 2009 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Wed, 12 Aug 2009 14:46:55 -0400 Subject: AO scalability issue In-Reply-To: References: Message-ID: <74e91d510908121146t364ede0fl186f25d7fcd887a8@mail.gmail.com> Hi All, I can take charge of the development of this. Ulisses, we can take this discussion offline. The PETSc team only takes patches against their current working version, but I can work against an earlier PETSc if that is preferred. Thanks, Aron On Wed, Aug 12, 2009 at 2:19 PM, Matthew Knepley wrote: > On Wed, Aug 12, 2009 at 1:01 PM, Lianjun An wrote: > >> We currently use Petsc routine "AOCreateBasic" to create application >> order --- a mapping from used id to petsc id. Then we setup a connection >> graph for our mesh and assemble vector and matrix based on the Petsc order. >> When total number of elements in our mesh is about couple million and 64 >> computer nodes are used, AO is created in a reasonable time. But when we >> have 700 million elements in the mesh and use more then 256 computer nodes, >> the time to create AO grows significantly (9000 secs). >> >> It seems that each process has whole permutation table about the mapping >> and quite expensive in time and buffer size. >> a) Memory requirement is quite high (AO creation failed for 1.3 billion >> elements on 1024 nodes due to not enough memory). >> b) If user id numbering on each processor is partially continuous, then >> time for creation AO is reduced. >> c) In my understanding, AOApplicationToPetsc and AOPetscToApplication >> routines don't use communication since AO has whole mapping on each >> processor. >> >> In our application, we might not need the whole mapping in each processor. >> We only need to ids of vertices that reside in the processor (limited ghost >> vertices). We wonder whether there is an efficient way to create AO and to >> cut its creation time and buffer size. Let us know your thought on that. >> > > The AO was not designed to be scalable (unfortunately). Any scalable > version would segment the renumbering, so > that each process was responsible for a range of entries. You would first > communicate the indices to correct process > (using a VecScatter), renumber, and then communicate the result back to the > original process. It does not sound all > that hard to me (using contiguous indices), however we are unlikely to > implement it right now. If you would like to > try yourself, we would be willing to help. > > Matt > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Aron Jamil Ahmadia Assistant Research Scientist King Abdullah University of Science and Technology -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.wornom at sophia.inria.fr Thu Aug 13 02:02:18 2009 From: stephen.wornom at sophia.inria.fr (Stephen Wornom) Date: Thu, 13 Aug 2009 09:02:18 +0200 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> <4A82E228.90400@sophia.inria.fr> Message-ID: <4A83BA7A.2060106@sophia.inria.fr> Matthew Knepley wrote: > On Wed, Aug 12, 2009 at 10:39 AM, Stephen Wornom > > wrote: > > Matthew Knepley wrote: > > 1) You should really handle this by creating the constant > vector on the pressure > space and using MatNullSpaceCreate() > > 2) You can also easily handle this by fixing the pressure at > one point > > At what indices or location does one fix the pressure? What value > is it set. Usually pressure is part of the solution. It would seem > to introduce an inconsistency. > I would like to understand how to do it. > > > It does not matter where you fix it or to what value. The absolute > value of the pressure is not physically > relevant, only pressure differences (which is why only grad p appears). Thanks for clarification on this point which is confusing to many of us. So why is it necessary to fix a value at a point since as you point out that only the grad p appears? If the pressure is not set in the numerical eqns, will you agree that the initial pressure sets the value for the numerical pressure at least in time accurate solutions. > Fixing the pressure at any point > to any value just sets the scale. There is no inconsistency. This is > in many many elementary fluid mechanics > books. Pass along a reference for myself and students. Thanks again, Stephen > > To do this mechanically. Change one row in the operator div (from the > div u = 0 equation) to the identity. > > Matt > > > > Stephen > > > Matt > > > On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins > > >> wrote: > > > Stephen, > > There are two ways that I know of to deal with pressure checker > boarding: staggered grids or some form of Rhie-Chow > interpolation. > IMO, these are simple only for uniform, Cartesian grids. > For grids > that are curvilinear, unstructured, non-uniform, and/or > non-orthogonal, things get real complicated. There may be other > methods, but something is required. > > Regarding boundary conditions, I would suggest this text book: > > H. K. Versteeg and W. Malalasekera. An Introduction to > Computational Fluid Dynamics, the Finite Volume Method. 2nd > edition. Prentice-Hall. 2007 > > While this book uses the finite volume method, the > explanation of > boundary conditions and staggered grids is very good and > relatively > easy to interpret for finite difference. I would also > recommend > > Joel H. Ferziger and Milovan Peric. Computational > Methods for > Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. > > This is a little more general with regard to method > discussing finite > difference and finite volume, but still settling on finite > volume. > > My $0.02: I question the use of finite difference. For > Navier-Stokes, > the use of finite volume is much more prevalent in > commercial and > research codes. If your student follows Versteeg and > Malalasekera a > simple, working, staggered grid FV code could be built in a > very short > time. If something more complicated is needed, it's probably > explained in Ferziger and Peric. > > Also My $0.02: Unless the point of your student's work is to > experience building her own code, why not download > something like > OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use > it? I > expect the effort to learn something like OpenFOAM for a simple > application will be much less than writing a new code. > > Hope this helps. > > Bill > > >>>>> "Stephen" == Stephen Wornom > > >> writes: > > Stephen> Shengyong wrote: > >> Hi, Farshid > >> > >> Maybe she should use the staggered grid method which > is very > simple to > >> implement. > Stephen> Does it remain simple for curvilinear meshes? > Stephen> Stephen > >> > >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby > > > > >> >>> wrote: > >> > >> Hi all, > >> > >> Sorry for this off-topic post. > >> > >> I am helping a master studnet which is working on solving > >> Navier-Stokes equation with Finite Difference method. > She is > >> trying to eliminate spourious pressure modes from the > solution. > >> She needs to know some details that are not usually > found in the > >> papers but important when programming, e.g. boundary > condition for > >> pressure. If someone has expertise on this or know a > *simple* FD > >> code, I would be thankful to let me know. > >> > >> Best regards, > >> Farshid Mossaiby > >> > >> > >> > >> > >> > >> > >> -- > >> Pang Shengyong > >> Solidification Simulation Lab, > >> State Key Lab of Mould & Die Technology, > >> Huazhong Univ. of Sci. & Tech. China > > > -- > Bill Perkins > Research Engineer > Hydrology Group > > Pacific Northwest National Laboratory > 902 Battelle Boulevard > P.O. Box 999, MSIN K9-36 > Richland, WA 99352 USA > Tel: 509-372-6131 > Fax: 509-372-6089 > william.perkins at pnl.gov > > > www.pnl.gov > > > > > > -- > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any > results to which their experiments lead. > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener From b.van-wachem at imperial.ac.uk Thu Aug 13 03:03:19 2009 From: b.van-wachem at imperial.ac.uk (Berend van Wachem) Date: Thu, 13 Aug 2009 09:03:19 +0100 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: <4A83BA7A.2060106@sophia.inria.fr> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> <4A82E228.90400@sophia.inria.fr> <4A83BA7A.2060106@sophia.inria.fr> Message-ID: <4A83C8C7.1030600@imperial.ac.uk> Dear Stephen, Farshid, Because you have an equation for grad p (or nabla^2 p for most pressure solvers), you need to specify a value for p somewhere in the problem: otherwise p is under-determined. (e.g. if grad p need to be 100, the values of p of 0 and 100 satisfy this, but also 1100 and 1000). You can specify the value p at a point, at a face, or at a boundary, depending on the problem you are looking at. If you want to develop a Navier-Stokes solver in terms of a few months, e.g. for an MSc project, I recommend you to do it in a staggered fashion. On of the best references, I think, still is Patankar, @BOOK{patankar80, title = {Numerical Heat Transfer and Fluid Flow}, publisher = {Hemisphere Publising Corporation}, year = {1980}, author = {S.V. Patankar} owner = {berend} } However, staggered gets messy for non-Cartesian domains/grids. Then I would recommend collocated - but this is really a lot more difficult than staggered. The pressure-velocity coupling algorithm is not straightforward and I see many mistakes in this, even in journal papers. For collocated grids, there is a nice chapter in Ferziger&Peric: @BOOK{ferziger02, title = {Computational Methods for Fluid Dynamics}, publisher = {Springer}, year = {2002}, author = {J.H. Ferziger and M. Peric}, owner = {berend} } Let me know if you have any specific questions. Best regards, Berend van Wachem. Stephen Wornom wrote: > Matthew Knepley wrote: >> On Wed, Aug 12, 2009 at 10:39 AM, Stephen Wornom >> > > wrote: >> >> Matthew Knepley wrote: >> >> 1) You should really handle this by creating the constant >> vector on the pressure >> space and using MatNullSpaceCreate() >> >> 2) You can also easily handle this by fixing the pressure at >> one point >> >> At what indices or location does one fix the pressure? What value >> is it set. Usually pressure is part of the solution. It would seem >> to introduce an inconsistency. >> I would like to understand how to do it. >> >> >> It does not matter where you fix it or to what value. The absolute >> value of the pressure is not physically >> relevant, only pressure differences (which is why only grad p appears). > Thanks for clarification on this point which is confusing to many of us. > So why is it necessary to fix a value at a point since as you point out > that only the grad p appears? If the pressure is not set in the > numerical eqns, will you agree that the initial pressure sets the value > for the numerical pressure at least in time accurate solutions. >> Fixing the pressure at any point >> to any value just sets the scale. There is no inconsistency. This is >> in many many elementary fluid mechanics >> books. > Pass along a reference for myself and students. > Thanks again, > Stephen >> To do this mechanically. Change one row in the operator div (from the >> div u = 0 equation) to the identity. >> >> Matt >> >> >> >> Stephen >> >> >> Matt >> >> >> On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins >> >> > >> wrote: >> >> >> Stephen, >> >> There are two ways that I know of to deal with pressure checker >> boarding: staggered grids or some form of Rhie-Chow >> interpolation. >> IMO, these are simple only for uniform, Cartesian grids. >> For grids >> that are curvilinear, unstructured, non-uniform, and/or >> non-orthogonal, things get real complicated. There may be other >> methods, but something is required. >> >> Regarding boundary conditions, I would suggest this text book: >> >> H. K. Versteeg and W. Malalasekera. An Introduction to >> Computational Fluid Dynamics, the Finite Volume Method. 2nd >> edition. Prentice-Hall. 2007 >> >> While this book uses the finite volume method, the >> explanation of >> boundary conditions and staggered grids is very good and >> relatively >> easy to interpret for finite difference. I would also >> recommend >> >> Joel H. Ferziger and Milovan Peric. Computational >> Methods for >> Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. >> >> This is a little more general with regard to method >> discussing finite >> difference and finite volume, but still settling on finite >> volume. >> >> My $0.02: I question the use of finite difference. For >> Navier-Stokes, >> the use of finite volume is much more prevalent in >> commercial and >> research codes. If your student follows Versteeg and >> Malalasekera a >> simple, working, staggered grid FV code could be built in a >> very short >> time. If something more complicated is needed, it's probably >> explained in Ferziger and Peric. >> >> Also My $0.02: Unless the point of your student's work is to >> experience building her own code, why not download >> something like >> OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use >> it? I >> expect the effort to learn something like OpenFOAM for a simple >> application will be much less than writing a new code. >> >> Hope this helps. >> >> Bill >> >> >>>>> "Stephen" == Stephen Wornom >> > >> > >> writes: >> >> Stephen> Shengyong wrote: >> >> Hi, Farshid >> >> >> >> Maybe she should use the staggered grid method which >> is very >> simple to >> >> implement. >> Stephen> Does it remain simple for curvilinear meshes? >> Stephen> Stephen >> >> >> >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby >> >> > >> >> > > >>> wrote: >> >> >> >> Hi all, >> >> >> >> Sorry for this off-topic post. >> >> >> >> I am helping a master studnet which is working on solving >> >> Navier-Stokes equation with Finite Difference method. >> She is >> >> trying to eliminate spourious pressure modes from the >> solution. >> >> She needs to know some details that are not usually >> found in the >> >> papers but important when programming, e.g. boundary >> condition for >> >> pressure. If someone has expertise on this or know a >> *simple* FD >> >> code, I would be thankful to let me know. >> >> >> >> Best regards, >> >> Farshid Mossaiby >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Pang Shengyong >> >> Solidification Simulation Lab, >> >> State Key Lab of Mould & Die Technology, >> >> Huazhong Univ. of Sci. & Tech. China >> >> >> -- >> Bill Perkins >> Research Engineer >> Hydrology Group >> >> Pacific Northwest National Laboratory >> 902 Battelle Boulevard >> P.O. Box 999, MSIN K9-36 >> Richland, WA 99352 USA >> Tel: 509-372-6131 >> Fax: 509-372-6089 >> william.perkins at pnl.gov >> > >> www.pnl.gov >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin >> their experiments is infinitely more interesting than any >> results to which their experiments lead. >> -- Norbert Wiener >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener > From knepley at gmail.com Thu Aug 13 06:30:42 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 13 Aug 2009 06:30:42 -0500 Subject: Off topic - Some advice on solving Navier-Stokes with FiniteDifference In-Reply-To: <4A83BA7A.2060106@sophia.inria.fr> References: <655552.9097.qm@web52211.mail.re2.yahoo.com> <4A82D21D.3070409@sophia.inria.fr> <19074.56683.949429.316119@bearflag.pnl.gov> <4A82E228.90400@sophia.inria.fr> <4A83BA7A.2060106@sophia.inria.fr> Message-ID: On Thu, Aug 13, 2009 at 2:02 AM, Stephen Wornom < stephen.wornom at sophia.inria.fr> wrote: > Matthew Knepley wrote: > >> On Wed, Aug 12, 2009 at 10:39 AM, Stephen Wornom < >> stephen.wornom at sophia.inria.fr > >> wrote: >> >> Matthew Knepley wrote: >> >> 1) You should really handle this by creating the constant >> vector on the pressure >> space and using MatNullSpaceCreate() >> >> 2) You can also easily handle this by fixing the pressure at >> one point >> >> At what indices or location does one fix the pressure? What value >> is it set. Usually pressure is part of the solution. It would seem >> to introduce an inconsistency. >> I would like to understand how to do it. >> >> >> It does not matter where you fix it or to what value. The absolute value >> of the pressure is not physically >> relevant, only pressure differences (which is why only grad p appears). >> > Thanks for clarification on this point which is confusing to many of us. So > why is it necessary to fix a value at a point since as you point out that > only the grad p appears? If the pressure is not set in the numerical eqns, > will you agree that the initial pressure sets the value for the numerical > pressure at least in time accurate solutions. > No! This is the fundamental mathematical difference between compressible and incompressible flow (see article by Weinan E). In compressible flow, pressure is a true dynamical variable, and this is why we need an equation of state to close the system. In incompressible flow, pressure is merely the Lagrange multiplier that enforces the divergence-free constraint. It is not a dynamical variable, and does not have initial conditions. > Fixing the pressure at any point >> to any value just sets the scale. There is no inconsistency. This is in >> many many elementary fluid mechanics >> books. >> > Pass along a reference for myself and students. There is a nice review of incompressible flow by Stefan Turek which goes over every solution method I have seen. > > Thanks again, > Stephen > >> >> To do this mechanically. Change one row in the operator div (from the div >> u = 0 equation) to the identity. >> >> Matt >> >> >> Stephen >> >> >> Matt >> >> >> On Wed, Aug 12, 2009 at 10:19 AM, William A. Perkins >> >> > >> wrote: >> >> >> Stephen, >> >> There are two ways that I know of to deal with pressure checker >> boarding: staggered grids or some form of Rhie-Chow >> interpolation. >> IMO, these are simple only for uniform, Cartesian grids. >> For grids >> that are curvilinear, unstructured, non-uniform, and/or >> non-orthogonal, things get real complicated. There may be other >> methods, but something is required. >> >> Regarding boundary conditions, I would suggest this text book: >> >> H. K. Versteeg and W. Malalasekera. An Introduction to >> Computational Fluid Dynamics, the Finite Volume Method. 2nd >> edition. Prentice-Hall. 2007 >> >> While this book uses the finite volume method, the >> explanation of >> boundary conditions and staggered grids is very good and >> relatively >> easy to interpret for finite difference. I would also >> recommend >> >> Joel H. Ferziger and Milovan Peric. Computational >> Methods for >> Fluid Dynamics. Springer-Verlag, 3rd edition, 2002. >> >> This is a little more general with regard to method >> discussing finite >> difference and finite volume, but still settling on finite >> volume. >> >> My $0.02: I question the use of finite difference. For >> Navier-Stokes, >> the use of finite volume is much more prevalent in >> commercial and >> research codes. If your student follows Versteeg and >> Malalasekera a >> simple, working, staggered grid FV code could be built in a >> very short >> time. If something more complicated is needed, it's probably >> explained in Ferziger and Peric. >> >> Also My $0.02: Unless the point of your student's work is to >> experience building her own code, why not download >> something like >> OpenFOAM (http://www.opencfd.co.uk/openfoam/) and just use >> it? I >> expect the effort to learn something like OpenFOAM for a simple >> application will be much less than writing a new code. >> >> Hope this helps. >> >> Bill >> >> >>>>> "Stephen" == Stephen Wornom >> > >> > >> writes: >> >> Stephen> Shengyong wrote: >> >> Hi, Farshid >> >> >> >> Maybe she should use the staggered grid method which >> is very >> simple to >> >> implement. >> Stephen> Does it remain simple for curvilinear meshes? >> Stephen> Stephen >> >> >> >> On Tue, Aug 11, 2009 at 5:16 AM, Farshid Mossaiby >> >> > >> >> > > >>> wrote: >> >> >> >> Hi all, >> >> >> >> Sorry for this off-topic post. >> >> >> >> I am helping a master studnet which is working on solving >> >> Navier-Stokes equation with Finite Difference method. >> She is >> >> trying to eliminate spourious pressure modes from the >> solution. >> >> She needs to know some details that are not usually >> found in the >> >> papers but important when programming, e.g. boundary >> condition for >> >> pressure. If someone has expertise on this or know a >> *simple* FD >> >> code, I would be thankful to let me know. >> >> >> >> Best regards, >> >> Farshid Mossaiby >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Pang Shengyong >> >> Solidification Simulation Lab, >> >> State Key Lab of Mould & Die Technology, >> >> Huazhong Univ. of Sci. & Tech. China >> >> >> -- >> Bill Perkins >> Research Engineer >> Hydrology Group >> >> Pacific Northwest National Laboratory >> 902 Battelle Boulevard >> P.O. Box 999, MSIN K9-36 >> Richland, WA 99352 USA >> Tel: 509-372-6131 >> Fax: 509-372-6089 >> william.perkins at pnl.gov >> > >> www.pnl.gov >> >> >> >> >> >> -- What most experimenters take for granted before they >> begin >> their experiments is infinitely more interesting than any >> results to which their experiments lead. >> -- Norbert Wiener >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From u.tabak at tudelft.nl Thu Aug 13 10:35:25 2009 From: u.tabak at tudelft.nl (Umut Tabak) Date: Thu, 13 Aug 2009 17:35:25 +0200 Subject: Slepc left eigenvectors Message-ID: <4A8432BD.8040307@tudelft.nl> Dear all, Is there a way to compute the left eigenvectors for a generalized problem using slepc? Best regards, Umut From jroman at dsic.upv.es Thu Aug 13 11:25:02 2009 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 13 Aug 2009 18:25:02 +0200 Subject: Slepc left eigenvectors In-Reply-To: <4A8432BD.8040307@tudelft.nl> References: <4A8432BD.8040307@tudelft.nl> Message-ID: <63C21918-DFAB-4A76-94E4-DF3313FE98B0@dsic.upv.es> On 13/08/2009, Umut Tabak wrote: > Dear all, > Is there a way to compute the left eigenvectors for a generalized > problem using slepc? > Best regards, > Umut Support for left eigenvectors is still under development. For the moment, one needs to solve an eigenproblem associated to the matrix pair (At,Bt), where At and Bt are explicit transposes obtained with MatTranspose. Please send any SLEPc-related questions to the SLEPc maintainance email (we read the PETSc users list only occasionally). Jose From likask at civil.gla.ac.uk Fri Aug 14 03:54:23 2009 From: likask at civil.gla.ac.uk (Lukasz Kaczmarczyk) Date: Fri, 14 Aug 2009 09:54:23 +0100 Subject: MatCreateMPIAIJWithSplitArrays and MatSetValues Message-ID: <242CA013-0D42-4E1B-A332-8C59883CBD13@civil.gla.ac.uk> Hello I created matrix with some off-processor entries using MatCreateMPIAIJWithSplitArrays. However I noticed problem, I can not set values on off-proc entries by using MatSetValue and MatSetValues. Can You help me? Regards Lukasz From knepley at gmail.com Fri Aug 14 10:08:01 2009 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 14 Aug 2009 10:08:01 -0500 Subject: MatCreateMPIAIJWithSplitArrays and MatSetValues In-Reply-To: <242CA013-0D42-4E1B-A332-8C59883CBD13@civil.gla.ac.uk> References: <242CA013-0D42-4E1B-A332-8C59883CBD13@civil.gla.ac.uk> Message-ID: On Fri, Aug 14, 2009 at 3:54 AM, Lukasz Kaczmarczyk wrote: > Hello > > I created matrix with some off-processor entries using > MatCreateMPIAIJWithSplitArrays. However I noticed problem, I can not set > values on off-proc entries by using MatSetValue and MatSetValues. Can You > help me? You should be able to (after calling MatAssemblyBegin/End). What happens? Can you give a simple example. Matt > > Regards > Lukasz > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Aug 14 11:35:15 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 14 Aug 2009 11:35:15 -0500 Subject: MatCreateMPIAIJWithSplitArrays and MatSetValues In-Reply-To: <242CA013-0D42-4E1B-A332-8C59883CBD13@civil.gla.ac.uk> References: <242CA013-0D42-4E1B-A332-8C59883CBD13@civil.gla.ac.uk> Message-ID: MatCreateMPIAIJWithSplitArrays() cannot be used to set some off- processor entries; it sets only local rows. Also you cannot later set off-processor values into this matrix with MatSetValues(). This is rather a special routine designed for people who already have the entire sparse matrix setup in parallel CSR format. For general usage where you set values with MatSetValues() you should use MatCreateMPIAIJ() Barry On Aug 14, 2009, at 3:54 AM, Lukasz Kaczmarczyk wrote: > Hello > > I created matrix with some off-processor entries using > MatCreateMPIAIJWithSplitArrays. However I noticed problem, I can not > set values on off-proc entries by using MatSetValue and > MatSetValues. Can You help me? > > Regards > Lukasz From z.sheng at ewi.tudelft.nl Mon Aug 17 09:58:12 2009 From: z.sheng at ewi.tudelft.nl (zhifeng sheng) Date: Mon, 17 Aug 2009 16:58:12 +0200 Subject: Introducing sources in ODE? In-Reply-To: <63C21918-DFAB-4A76-94E4-DF3313FE98B0@dsic.upv.es> References: <4A8432BD.8040307@tudelft.nl> <63C21918-DFAB-4A76-94E4-DF3313FE98B0@dsic.upv.es> Message-ID: <4A897004.5090703@ewi.tudelft.nl> Dear all I would like to use Time-steppers - ODE integrators - TS to solve my time domain problem... However, the equation has the form Au(t) + f(t) = u'(t) ... and by looking into the example code, I could not find a way to introduce the source term f(t)... Does anyone know how to do that? If you do, please help with this. Thanks and best regards Zhifeng From nicolas.aunai at gmail.com Tue Aug 18 02:30:38 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Tue, 18 Aug 2009 09:30:38 +0200 Subject: linear elliptic vector equation Message-ID: Dear all, I have a simulation code (particle in cell) in which I have to solve a linear vector equation 4 times per time step. A typical run consists of 50 000 time steps. The grid is rectangular and uniform of size Lx, Ly, with nx+1 and ny+1 points in the x and y direction respectively., (nx, ny) could be at max (1024,1024). The vector equation is the following : B(x,y) - a*Laplacian(B(x,y)) = S(x,y) Where B and S are 2D vector fields with 3 components (Bx, By, Bz and Sx, Sy, Sz), each depending on the x and y coordinates. 'a' is a positive constant, smaller than one, typically a= 0.02 Boundary conditions may depend on for which B component we are solving the equation. On the x=cst borders, the boundary is always periodic, no matter what component is solved, but on y=cst borders, the boundary condition can be either Neumann or Dirichlet. So far, I have written a small code that creates a vector solution, a vector RHS and the matrix operator, and solve a scalar equation of this type. Solving my vector equation would then just call 3 times this kind of code... but I believe there is another way to deal with vector fields and linear systems with Petsc, using DAs no ? Could someone explain me how solve this the proper way and/or show me some code solving linear system with vector fields ? (is there an example in the exercices that I would not have seen ?) Thanks a lot Nico From aron.ahmadia at kaust.edu.sa Tue Aug 18 08:26:34 2009 From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia) Date: Tue, 18 Aug 2009 09:26:34 -0400 Subject: linear elliptic vector equation In-Reply-To: References: Message-ID: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> ex19 comes to mind, though it's a bit overkill for what you're doing... http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html ex19 uses DAs and DMMG, which is kind of like a meta-DA for using multigrid-style solvers. Both work well with structured grids. You may also want to review the PETSc manual section on DA. A On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai wrote: > Dear all, > > > I have a simulation code (particle in cell) in which I have to solve a > linear vector equation 4 times per time step. A typical run consists > of 50 000 time steps. The grid is rectangular and uniform of size Lx, > Ly, with nx+1 and ny+1 points in the x and y direction respectively., > (nx, ny) could be at max (1024,1024). > > The vector equation is the following : > > B(x,y) - a*Laplacian(B(x,y)) = S(x,y) > > > Where B and S are 2D vector fields with 3 components (Bx, By, Bz and > Sx, Sy, Sz), each depending on the x and y coordinates. > > > 'a' is a positive constant, smaller than one, typically a= 0.02 > > Boundary conditions may depend on for which B component we are solving > the equation. On the x=cst borders, the boundary is always periodic, > no matter what component is solved, but on y=cst borders, the boundary > condition can be either Neumann or Dirichlet. > > So far, I have written a small code that creates a vector solution, a > vector RHS and the matrix operator, and solve a scalar equation of > this type. Solving my vector equation would then just call 3 times > this kind of code... but I believe there is another way to deal with > vector fields and linear systems with Petsc, using DAs no ? Could > someone explain me how solve this the proper way and/or show me some > code solving linear system with vector fields ? (is there an example > in the exercices that I would not have seen ?) > > > > Thanks a lot > Nico > -- Aron Jamil Ahmadia Assistant Research Scientist King Abdullah University of Science and Technology -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.aunai at gmail.com Tue Aug 18 11:42:42 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Tue, 18 Aug 2009 18:42:42 +0200 Subject: linear elliptic vector equation In-Reply-To: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> Message-ID: Hello, Yes it is a bit overkill indeed :-) I was looking at ex29 actually... which is still linear and with DA's but scalar. But I'd rather like not use multigrids, however I have not seen an example of a code that uses DA's and KSP solvers. Every example use multigrids. When you have a multicomponent linear equation to solve on a rectangular uniform grid such as mine, is it good to : 1/ use DA's to represent your grid 2/ create the matrix corresponding to your operator once and for all (the operator neve changes).( In fact there are 2 matrices because two cases possibles (Neumann/periodic or Dirichlet/Periodic, depending on the component). but never minds) 3/ Call KSPSolve() 3 times (for each vector component associated with the DA) ? Thx Nico 2009/8/18 Aron Ahmadia : > ex19 comes to mind, though it's a bit overkill for what you're doing... > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html > > ex19 uses DAs and DMMG, which is kind of like a meta-DA for using > multigrid-style solvers.? Both work well with structured grids. > > You may also want to review the PETSc manual section on DA. > > A > > On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai > wrote: >> >> Dear all, >> >> >> I have a simulation code (particle in cell) in which I have to solve a >> linear vector equation 4 times per time step. A typical run consists >> of 50 000 time steps. The grid is rectangular and uniform of size Lx, >> Ly, with nx+1 and ny+1 points in the x and y direction respectively., >> (nx, ny) could be at max (1024,1024). >> >> The vector equation is the following : >> >> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) >> >> >> Where B and S are 2D vector fields with 3 components (Bx, By, Bz and >> Sx, Sy, Sz), each depending on the x and y coordinates. >> >> >> 'a' is a positive constant, smaller than one, typically a= 0.02 >> >> Boundary conditions may depend on for which B component we are solving >> the equation. On the x=cst borders, the boundary is always periodic, >> no matter what component is solved, but on y=cst borders, the boundary >> condition can be either Neumann or Dirichlet. >> >> So far, I have written a small code that creates a vector solution, a >> vector RHS and the matrix operator, and solve a scalar equation of >> this type. Solving my vector equation would then just call 3 times >> this kind of code... but I believe there is another way to deal with >> vector fields and linear systems with Petsc, using DAs no ? Could >> someone explain me how solve this the proper way and/or show me some >> code solving linear system with vector fields ? (is there an example >> in the exercices that I would not have seen ?) >> >> >> >> Thanks a lot >> Nico > > > > -- > Aron Jamil Ahmadia > Assistant Research Scientist > King Abdullah University of Science and Technology > From bsmith at mcs.anl.gov Tue Aug 18 21:41:54 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Aug 2009 21:41:54 -0500 Subject: linear elliptic vector equation In-Reply-To: References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> Message-ID: <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> On Aug 18, 2009, at 11:42 AM, nicolas aunai wrote: > Hello, > > Yes it is a bit overkill indeed :-) I was looking at ex29 actually... > which is still linear and with DA's but scalar. > > But I'd rather like not use multigrids, however I have not seen an > example of a code that uses DA's and KSP solvers. > > Every example use multigrids. Multigrid is the way to go for this problem, it will be faster than some generic preconditioned CG. > > When you have a multicomponent linear equation to solve on a > rectangular uniform grid such as mine, is it good to : > > 1/ use DA's to represent your grid > > 2/ create the matrix corresponding to your operator once and for all > (the operator neve changes).( In fact there are 2 matrices because two > cases possibles (Neumann/periodic or Dirichlet/Periodic, depending on > the component). but never minds) > > > 3/ Call KSPSolve() 3 times (for each vector component associated > with the DA) ? > Yes. You can do all this using the DMMG object. Note that the DMMG object in your linear case just manages a KSP and the meshes for you. In the end it is a KSPSolve() that actually does the solve and you have full control over the solve including using -pc_type icc or something else that does not use multigrid. Though multigrid will be faster. Barry > > Thx > Nico > > > 2009/8/18 Aron Ahmadia : >> ex19 comes to mind, though it's a bit overkill for what you're >> doing... >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html >> >> ex19 uses DAs and DMMG, which is kind of like a meta-DA for using >> multigrid-style solvers. Both work well with structured grids. >> >> You may also want to review the PETSc manual section on DA. >> >> A >> >> On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai > > >> wrote: >>> >>> Dear all, >>> >>> >>> I have a simulation code (particle in cell) in which I have to >>> solve a >>> linear vector equation 4 times per time step. A typical run consists >>> of 50 000 time steps. The grid is rectangular and uniform of size >>> Lx, >>> Ly, with nx+1 and ny+1 points in the x and y direction >>> respectively., >>> (nx, ny) could be at max (1024,1024). >>> >>> The vector equation is the following : >>> >>> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) >>> >>> >>> Where B and S are 2D vector fields with 3 components (Bx, By, Bz and >>> Sx, Sy, Sz), each depending on the x and y coordinates. >>> >>> >>> 'a' is a positive constant, smaller than one, typically a= 0.02 >>> >>> Boundary conditions may depend on for which B component we are >>> solving >>> the equation. On the x=cst borders, the boundary is always periodic, >>> no matter what component is solved, but on y=cst borders, the >>> boundary >>> condition can be either Neumann or Dirichlet. >>> >>> So far, I have written a small code that creates a vector >>> solution, a >>> vector RHS and the matrix operator, and solve a scalar equation of >>> this type. Solving my vector equation would then just call 3 times >>> this kind of code... but I believe there is another way to deal with >>> vector fields and linear systems with Petsc, using DAs no ? Could >>> someone explain me how solve this the proper way and/or show me some >>> code solving linear system with vector fields ? (is there an example >>> in the exercices that I would not have seen ?) >>> >>> >>> >>> Thanks a lot >>> Nico >> >> >> >> -- >> Aron Jamil Ahmadia >> Assistant Research Scientist >> King Abdullah University of Science and Technology >> From bsmith at mcs.anl.gov Tue Aug 18 21:52:00 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 18 Aug 2009 21:52:00 -0500 Subject: Introducing sources in ODE? In-Reply-To: <4A897004.5090703@ewi.tudelft.nl> References: <4A8432BD.8040307@tudelft.nl> <63C21918-DFAB-4A76-94E4-DF3313FE98B0@dsic.upv.es> <4A897004.5090703@ewi.tudelft.nl> Message-ID: <7494BF6B-E96D-4331-B3E9-9547F325215F@mcs.anl.gov> I'm sorry, we never thought to add this support to the "linear case". You could use the nonlinear TS interface to solve your problem, but that is a bit overkill since it is only linear in U. What particular integrator are you interested in using? Perhaps you could add it yourself once you understand the source code in TS. Or perhaps someone else has time to try to add this support? Me, I'd just try doing it with the nonlinear interface where the entire Au(t) + f(t) is the nonlinear RHS you provide and A is the Jacobian you provide. Barry On Aug 17, 2009, at 9:58 AM, zhifeng sheng wrote: > Dear all > > I would like to use Time-steppers - ODE integrators - TS to solve my > time domain problem... > > However, the equation has the form Au(t) + f(t) = u'(t) ... and by > looking into the example code, I could not find a way to introduce > the source term f(t)... Does anyone know how to do that? If you do, > please help with this. > > Thanks and best regards > Zhifeng From nicolas.aunai at gmail.com Wed Aug 19 02:04:19 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Wed, 19 Aug 2009 09:04:19 +0200 Subject: linear elliptic vector equation In-Reply-To: <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> Message-ID: Hi, OK then, lets go for multigrid. When I'm looking for example at exercice 32 : http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex32.c.html I realize that I don't understand some things, in particular about the DMMGSetKSP function. I need to give the RHS function and the ComputeMatrix function. 1/ My operator is always the same during the simulation, I was thinking to build the matrix once and for all at the begining of the program. Why can't I do that ? Is it because at each multigrid level the required matrix is different (size) and it can only be built with the information the DA is giving ? 2/ I don't really understand the prototype of the callback function ComputeMatrix. The first argument is the DMMG object, ok. But the second and third arguments are matrices called 'J' and 'jac'. It seems (at least in ex32) that 'J' is not used, what is the goal of this matrix in general ? Is 'jac' stands for jacobian ? Reading the ComputeMatrix() code, it seems that the 'jac' matrix is built like a standard finite difference matrix, with the stencil corresponding to the operator I have to solve. if so, why is it called 'jacobian' ? (sorry it looks more a math question than a petsc question...) Another quick question : Why is there no documentation for the function 'DMMGGetr(dmmg)' which seems to be part of the petsc library since I can't see the definition of the function in the exercice file. I suppose it computes the residual ? I guess this is it.... for the moment. Thx Nico 2009/8/19 Barry Smith : > > On Aug 18, 2009, at 11:42 AM, nicolas aunai wrote: > >> Hello, >> >> Yes it is a bit overkill indeed :-) I was looking at ex29 actually... >> which is still linear and with DA's but scalar. >> >> But I'd rather like not use multigrids, however I have not seen an >> example of a code that uses DA's and KSP solvers. >> >> Every example use multigrids. > > ? Multigrid is the way to go for this problem, it will be faster than some > generic preconditioned CG. > >> >> When you have a multicomponent linear equation to solve on a >> rectangular uniform grid such as mine, is it good to : >> >> 1/ ?use DA's to represent your grid >> >> 2/ create the matrix corresponding to your operator once and for all >> (the operator neve changes).( In fact there are 2 matrices because two >> cases possibles (Neumann/periodic or Dirichlet/Periodic, depending on >> the component). but never minds) >> >> >> 3/ Call KSPSolve() 3 times (for each vector component associated with the >> DA) ? >> > ? Yes. You can do all this using the DMMG object. > > ?Note that the DMMG object in your linear case just manages a KSP and the > meshes for you. In the end it is a KSPSolve() that actually does the solve > and you have full control over the solve including using -pc_type icc or > something else that does not use multigrid. Though multigrid will be faster. > > > ? Barry > > >> >> Thx >> Nico >> >> >> 2009/8/18 Aron Ahmadia : >>> >>> ex19 comes to mind, though it's a bit overkill for what you're doing... >>> >>> >>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html >>> >>> ex19 uses DAs and DMMG, which is kind of like a meta-DA for using >>> multigrid-style solvers. ?Both work well with structured grids. >>> >>> You may also want to review the PETSc manual section on DA. >>> >>> A >>> >>> On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai >>> wrote: >>>> >>>> Dear all, >>>> >>>> >>>> I have a simulation code (particle in cell) in which I have to solve a >>>> linear vector equation 4 times per time step. A typical run consists >>>> of 50 000 time steps. The grid is rectangular and uniform of size Lx, >>>> Ly, with nx+1 and ny+1 points in the x and y direction respectively., >>>> (nx, ny) could be at max (1024,1024). >>>> >>>> The vector equation is the following : >>>> >>>> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) >>>> >>>> >>>> Where B and S are 2D vector fields with 3 components (Bx, By, Bz and >>>> Sx, Sy, Sz), each depending on the x and y coordinates. >>>> >>>> >>>> 'a' is a positive constant, smaller than one, typically a= 0.02 >>>> >>>> Boundary conditions may depend on for which B component we are solving >>>> the equation. On the x=cst borders, the boundary is always periodic, >>>> no matter what component is solved, but on y=cst borders, the boundary >>>> condition can be either Neumann or Dirichlet. >>>> >>>> So far, I have written a small code that creates a vector solution, a >>>> vector RHS and the matrix operator, and solve a scalar equation of >>>> this type. Solving my vector equation would then just call 3 times >>>> this kind of code... but I believe there is another way to deal with >>>> vector fields and linear systems with Petsc, using DAs no ? Could >>>> someone explain me how solve this the proper way and/or show me some >>>> code solving linear system with vector fields ? (is there an example >>>> in the exercices that I would not have seen ?) >>>> >>>> >>>> >>>> Thanks a lot >>>> Nico >>> >>> >>> >>> -- >>> Aron Jamil Ahmadia >>> Assistant Research Scientist >>> King Abdullah University of Science and Technology >>> > > From fredrik.bengzon at math.umu.se Wed Aug 19 04:53:07 2009 From: fredrik.bengzon at math.umu.se (Fredrik Bengzon) Date: Wed, 19 Aug 2009 11:53:07 +0200 Subject: cast PetscScalar to double Message-ID: <4A8BCB83.2000104@math.umu.se> Hi I'm using a Petsc 3 installation compiled with support for complex numbers. However, in the code I have made frequent use of the stl library and the standard type double. This now gives a compilation error since PetscScalar can not be casted into double. Do you know of a quick workaround or do I have to replace every instance of double with PetscScalar? Regards, Fredrik Bengzon From balay at mcs.anl.gov Wed Aug 19 08:17:59 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 19 Aug 2009 08:17:59 -0500 (CDT) Subject: cast PetscScalar to double In-Reply-To: <4A8BCB83.2000104@math.umu.se> References: <4A8BCB83.2000104@math.umu.se> Message-ID: you can replace (double)(var) with one of the following [approprate to your code] thingy. PetscAbsScalar(var) PetscRealPart(var) Satish On Wed, 19 Aug 2009, Fredrik Bengzon wrote: > Hi > I'm using a Petsc 3 installation compiled with support for complex numbers. > However, in the code I have made frequent use of the stl library and the > standard type double. This now gives a compilation error since PetscScalar can > not be casted into double. Do you know of a quick workaround or do I have to > replace every instance of double with PetscScalar? > Regards, > Fredrik Bengzon > From bsmith at mcs.anl.gov Wed Aug 19 13:01:10 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 19 Aug 2009 13:01:10 -0500 Subject: linear elliptic vector equation In-Reply-To: References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> Message-ID: <935B6944-C0D3-4F60-A02E-3A72A597DF8A@mcs.anl.gov> On Aug 19, 2009, at 2:04 AM, nicolas aunai wrote: > Hi, > > OK then, lets go for multigrid. > > > When I'm looking for example at exercice 32 : > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex32.c.html > > I realize that I don't understand some things, in particular about the > DMMGSetKSP function. > > I need to give the RHS function and the ComputeMatrix function. > > 1/ My operator is always the same during the simulation, I was > thinking to build the matrix once and for all at the begining of the > program. Why can't I do that ? Is it because at each multigrid level > the required matrix is different (size) and it can only be built with > the information the DA is giving ? Yes. The idea is rather than you giving the matrix. You give the function that fills up the matrix. > > 2/ I don't really understand the prototype of the callback function > ComputeMatrix. The first argument is the DMMG object, ok. But the > second and third arguments are matrices called 'J' and 'jac'. It > seems (at least in ex32) that 'J' is not used, what is the goal of > this matrix in general ? Sometimes one uses say a second order discretization to define the system but wants a cheap preconditioner built from the first order discretization. Hence the two matrices. If you are not doing this then you pass the same matrix in both slots. > > Is 'jac' stands for jacobian ? Reading the ComputeMatrix() code, it > seems that the 'jac' matrix is built like a standard finite difference > matrix, with the stencil corresponding to the operator I have to > solve. if so, why is it called 'jacobian' ? (sorry it looks more a > math question than a petsc question...) This is just brought over from the nonlinear solver code. It could just be called matrix here. Barry > > > Another quick question : > > Why is there no documentation for the function 'DMMGGetr(dmmg)' which > seems to be part of the petsc library since I can't see the definition > of the function in the exercice file. I suppose it computes the > residual ? > > > I guess this is it.... for the moment. > Thx > > Nico > > > > 2009/8/19 Barry Smith : >> >> On Aug 18, 2009, at 11:42 AM, nicolas aunai wrote: >> >>> Hello, >>> >>> Yes it is a bit overkill indeed :-) I was looking at ex29 >>> actually... >>> which is still linear and with DA's but scalar. >>> >>> But I'd rather like not use multigrids, however I have not seen an >>> example of a code that uses DA's and KSP solvers. >>> >>> Every example use multigrids. >> >> Multigrid is the way to go for this problem, it will be faster >> than some >> generic preconditioned CG. >> >>> >>> When you have a multicomponent linear equation to solve on a >>> rectangular uniform grid such as mine, is it good to : >>> >>> 1/ use DA's to represent your grid >>> >>> 2/ create the matrix corresponding to your operator once and for all >>> (the operator neve changes).( In fact there are 2 matrices because >>> two >>> cases possibles (Neumann/periodic or Dirichlet/Periodic, depending >>> on >>> the component). but never minds) >>> >>> >>> 3/ Call KSPSolve() 3 times (for each vector component associated >>> with the >>> DA) ? >>> >> Yes. You can do all this using the DMMG object. >> >> Note that the DMMG object in your linear case just manages a KSP >> and the >> meshes for you. In the end it is a KSPSolve() that actually does >> the solve >> and you have full control over the solve including using -pc_type >> icc or >> something else that does not use multigrid. Though multigrid will >> be faster. >> >> >> Barry >> >> >>> >>> Thx >>> Nico >>> >>> >>> 2009/8/18 Aron Ahmadia : >>>> >>>> ex19 comes to mind, though it's a bit overkill for what you're >>>> doing... >>>> >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html >>>> >>>> ex19 uses DAs and DMMG, which is kind of like a meta-DA for using >>>> multigrid-style solvers. Both work well with structured grids. >>>> >>>> You may also want to review the PETSc manual section on DA. >>>> >>>> A >>>> >>>> On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai >>> > >>>> wrote: >>>>> >>>>> Dear all, >>>>> >>>>> >>>>> I have a simulation code (particle in cell) in which I have to >>>>> solve a >>>>> linear vector equation 4 times per time step. A typical run >>>>> consists >>>>> of 50 000 time steps. The grid is rectangular and uniform of >>>>> size Lx, >>>>> Ly, with nx+1 and ny+1 points in the x and y direction >>>>> respectively., >>>>> (nx, ny) could be at max (1024,1024). >>>>> >>>>> The vector equation is the following : >>>>> >>>>> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) >>>>> >>>>> >>>>> Where B and S are 2D vector fields with 3 components (Bx, By, Bz >>>>> and >>>>> Sx, Sy, Sz), each depending on the x and y coordinates. >>>>> >>>>> >>>>> 'a' is a positive constant, smaller than one, typically a= 0.02 >>>>> >>>>> Boundary conditions may depend on for which B component we are >>>>> solving >>>>> the equation. On the x=cst borders, the boundary is always >>>>> periodic, >>>>> no matter what component is solved, but on y=cst borders, the >>>>> boundary >>>>> condition can be either Neumann or Dirichlet. >>>>> >>>>> So far, I have written a small code that creates a vector >>>>> solution, a >>>>> vector RHS and the matrix operator, and solve a scalar equation of >>>>> this type. Solving my vector equation would then just call 3 times >>>>> this kind of code... but I believe there is another way to deal >>>>> with >>>>> vector fields and linear systems with Petsc, using DAs no ? Could >>>>> someone explain me how solve this the proper way and/or show me >>>>> some >>>>> code solving linear system with vector fields ? (is there an >>>>> example >>>>> in the exercices that I would not have seen ?) >>>>> >>>>> >>>>> >>>>> Thanks a lot >>>>> Nico >>>> >>>> >>>> >>>> -- >>>> Aron Jamil Ahmadia >>>> Assistant Research Scientist >>>> King Abdullah University of Science and Technology >>>> >> >> From hal.finkel at yale.edu Wed Aug 19 22:54:03 2009 From: hal.finkel at yale.edu (Hal Finkel) Date: Wed, 19 Aug 2009 23:54:03 -0400 Subject: Matrix-free multigrid Message-ID: <4A8CC8DB.1000202@yale.edu> Hello, I'd like to try using a matrix-free Newton method with a multigrid object configured with -dmmg_grid_sequence, but I don't understand how. Using SNES directly, I can set -snes_mf, but this does not seems to work with the multigrid object, and DMMGSetSNES() always allocates a matrix. Thank you in advance, Hal From Stephen.R.Ball at awe.co.uk Thu Aug 20 08:48:03 2009 From: Stephen.R.Ball at awe.co.uk (Stephen Ball) Date: Thu, 20 Aug 2009 14:48:03 +0100 Subject: Update on PETSc GPU Message-ID: <98KEoU018964@awe.co.uk> Hi We are currently looking at procuring a GPU cluster for testing in the near future. I would be very keen to test your PETSc GPU when it is available. Can you give me an update on when you plan to make PETSc GPU available? Regards Stephen R. Ball This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal. From nicolas.aunai at gmail.com Thu Aug 20 13:27:05 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Thu, 20 Aug 2009 20:27:05 +0200 Subject: linear elliptic vector equation In-Reply-To: <935B6944-C0D3-4F60-A02E-3A72A597DF8A@mcs.anl.gov> References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> <935B6944-C0D3-4F60-A02E-3A72A597DF8A@mcs.anl.gov> Message-ID: Hi, I've written a little code based on the exercice 29, to solve my equation (u - mu*laplacian(u) = f) with one DOF and with dirichlet boundary conditions. my code is here : http://nau.cetp.ipsl.fr/inertia1.c It seems that the code is *very* slow (never waited the end...) when the dmmg_nlevels is > 4 (with a 3x3 coarsest grid). The refinement factor has the default value 2, and I want my fine grid to be for example nx=513, ny=257 grid points. I have some more questions : 1/ How can I change the linear solver and preconditionner *in* the code, not with the command -ksp_type and -pc_type ? 2/ What and Where is actually used the linear solver and the preconditionners ? Is it used at each grid level as a smoother ? Can we change the pre/post smoothing parameters ? 3/ My code is written to work in parallel, can I use a direct solver when I'm at the coarset grid (small matrix) ? 4/ When I add -ksp_monitor, do I see the residual of the smoothing iterations at each grid level ? If no, what is it ? 5/ My code is just a test code with some analytical RHS. In the end, the solver is supposed to be added in a simulation code where the RHS will be a/ the result of some other calculation, b/ changing with time (the system being solved at each time step). Is it relevant to look at the timing with this test code since the RHS is not the one that is supposed to be (i.e. is the RHS very important for convergence speed) ? 6/ When the solver will be added in the simulation, will it be much more efficient to setup the solution vector with the previous solution as an initial guess ? 7/ My goal is not to do full dirichlet conditions like I have done in the test code above, but rather DA_XPERIODIC and dirichlet in the Y direction. I have observed that with a refinement factor of 2, and a coarsest grid (nx=3,ny=3), when dmmg_nlevels is set to 2 I don't have a 5x5 grid but a 6x5 grid, why ? for dmmg_nlevels = 8 I have 384x257 grid instead of the 257^2 grid I was expected... what is the reason for this ? 8/ With DA_XPERIODIC, am I supposed to use the stencil as if I were within the domain when I'm on periodic "borders", i.e. is col[1].i = i-1 is ok when i=0 (left border) ? Ok that's it for the moment, Thx Nico 2009/8/19 Barry Smith : > > On Aug 19, 2009, at 2:04 AM, nicolas aunai wrote: > >> Hi, >> >> OK then, lets go for multigrid. >> >> >> When I'm looking for example at exercice 32 : >> >> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex32.c.html >> >> I realize that I don't understand some things, in particular about the >> DMMGSetKSP function. >> >> I need to give the RHS function and the ComputeMatrix function. >> >> 1/ My operator is always the same during the simulation, I was >> thinking to build the matrix once and for all at the begining of the >> program. Why can't I do that ? Is it because at each multigrid level >> the required matrix is different (size) and it can only be built with >> the information the DA is giving ? > > ? ?Yes. The idea is rather than you giving the matrix. You give the function > that fills up the matrix. >> >> 2/ I don't really understand the prototype of the callback function >> ComputeMatrix. The first argument is the DMMG object, ok. But the >> second and third arguments are matrices called 'J' and 'jac'. ?It >> seems (at least in ex32) that 'J' is not used, what is the goal of >> this matrix in general ? > > ? Sometimes one uses say a second order discretization to define the system > but wants a cheap preconditioner built from the first order discretization. > Hence the two matrices. If you are not doing this then you pass the same > matrix in both slots. > >> >> Is 'jac' stands for jacobian ? Reading the ComputeMatrix() code, it >> seems that the 'jac' matrix is built like a standard finite difference >> matrix, with the stencil corresponding to the operator I have to >> solve. if so, why is it called 'jacobian' ? (sorry it looks more a >> math question than a petsc question...) > > ? This is just brought over from the nonlinear solver code. It could just be > called matrix here. > > ? Barry > >> >> >> Another quick question : >> >> Why is there no documentation for the function 'DMMGGetr(dmmg)' which >> seems to be part of the petsc library since I can't see the definition >> of the function in the exercice file. I suppose it computes the >> residual ? >> >> >> I guess this is it.... for the moment. >> Thx >> >> Nico >> >> >> >> 2009/8/19 Barry Smith : >>> >>> On Aug 18, 2009, at 11:42 AM, nicolas aunai wrote: >>> >>>> Hello, >>>> >>>> Yes it is a bit overkill indeed :-) I was looking at ex29 actually... >>>> which is still linear and with DA's but scalar. >>>> >>>> But I'd rather like not use multigrids, however I have not seen an >>>> example of a code that uses DA's and KSP solvers. >>>> >>>> Every example use multigrids. >>> >>> ?Multigrid is the way to go for this problem, it will be faster than some >>> generic preconditioned CG. >>> >>>> >>>> When you have a multicomponent linear equation to solve on a >>>> rectangular uniform grid such as mine, is it good to : >>>> >>>> 1/ ?use DA's to represent your grid >>>> >>>> 2/ create the matrix corresponding to your operator once and for all >>>> (the operator neve changes).( In fact there are 2 matrices because two >>>> cases possibles (Neumann/periodic or Dirichlet/Periodic, depending on >>>> the component). but never minds) >>>> >>>> >>>> 3/ Call KSPSolve() 3 times (for each vector component associated with >>>> the >>>> DA) ? >>>> >>> ?Yes. You can do all this using the DMMG object. >>> >>> ?Note that the DMMG object in your linear case just manages a KSP and the >>> meshes for you. In the end it is a KSPSolve() that actually does the >>> solve >>> and you have full control over the solve including using -pc_type icc or >>> something else that does not use multigrid. Though multigrid will be >>> faster. >>> >>> >>> ?Barry >>> >>> >>>> >>>> Thx >>>> Nico >>>> >>>> >>>> 2009/8/18 Aron Ahmadia : >>>>> >>>>> ex19 comes to mind, though it's a bit overkill for what you're doing... >>>>> >>>>> >>>>> >>>>> http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html >>>>> >>>>> ex19 uses DAs and DMMG, which is kind of like a meta-DA for using >>>>> multigrid-style solvers. ?Both work well with structured grids. >>>>> >>>>> You may also want to review the PETSc manual section on DA. >>>>> >>>>> A >>>>> >>>>> On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai >>>>> >>>>> wrote: >>>>>> >>>>>> Dear all, >>>>>> >>>>>> >>>>>> I have a simulation code (particle in cell) in which I have to solve a >>>>>> linear vector equation 4 times per time step. A typical run consists >>>>>> of 50 000 time steps. The grid is rectangular and uniform of size Lx, >>>>>> Ly, with nx+1 and ny+1 points in the x and y direction respectively., >>>>>> (nx, ny) could be at max (1024,1024). >>>>>> >>>>>> The vector equation is the following : >>>>>> >>>>>> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) >>>>>> >>>>>> >>>>>> Where B and S are 2D vector fields with 3 components (Bx, By, Bz and >>>>>> Sx, Sy, Sz), each depending on the x and y coordinates. >>>>>> >>>>>> >>>>>> 'a' is a positive constant, smaller than one, typically a= 0.02 >>>>>> >>>>>> Boundary conditions may depend on for which B component we are solving >>>>>> the equation. On the x=cst borders, the boundary is always periodic, >>>>>> no matter what component is solved, but on y=cst borders, the boundary >>>>>> condition can be either Neumann or Dirichlet. >>>>>> >>>>>> So far, I have written a small code that creates a vector solution, a >>>>>> vector RHS and the matrix operator, and solve a scalar equation of >>>>>> this type. Solving my vector equation would then just call 3 times >>>>>> this kind of code... but I believe there is another way to deal with >>>>>> vector fields and linear systems with Petsc, using DAs no ? Could >>>>>> someone explain me how solve this the proper way and/or show me some >>>>>> code solving linear system with vector fields ? (is there an example >>>>>> in the exercices that I would not have seen ?) >>>>>> >>>>>> >>>>>> >>>>>> Thanks a lot >>>>>> Nico >>>>> >>>>> >>>>> >>>>> -- >>>>> Aron Jamil Ahmadia >>>>> Assistant Research Scientist >>>>> King Abdullah University of Science and Technology >>>>> >>> >>> > > From enjoywm at cs.wm.edu Fri Aug 21 10:01:18 2009 From: enjoywm at cs.wm.edu (Yixun Liu) Date: Fri, 21 Aug 2009 11:01:18 -0400 Subject: renumbering in petsc/src/dm/ao/examples/tutorials/ex2.c Message-ID: <4A8EB6BE.9000205@cs.wm.edu> Hi, I want to refine the renumbering method in ex2.c. Firstly determine the inner nodes and boundary nodes for each processors and then distribute boundary nodes to different processors according to the number of the inner nodes in the processor. For example, if the processors holds more inner nodes, it will receive few boundary nodes for the consideration of load balancing. At last renumber the nodes for each processor. I am not sure if this is a good method. Is there optimal method for renumbering? Thanks a lot. Yixun From cmay at phys.ethz.ch Sun Aug 23 05:52:23 2009 From: cmay at phys.ethz.ch (Christian May) Date: Sun, 23 Aug 2009 12:52:23 +0200 (CEST) Subject: hints wanted for huge eigenvalue problem Message-ID: Dear all, I want to extract the lowest eigenvalues of a huge generalized eigenvalue problem with very dense clusters of eigenvalues. I use slepc and superlu as external direct solver with the following options -eps_ncv 300 -eps_nev 220 -eps_tol 1e-10 -st_ksp_rtol 1e-14 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package superlu_dist Without shift-and-invert, this simply takes way too long, even on 48 CPUs it runs for hours. If I use a shift-and-invert technique additionally invoking -st_type sinvert -st_shift -0.41, it converges very fast and also parallelizes well, but I most likely don't get the lowest eigenvalues if st_shift is slightly too high. If it's slightly too low, it doesn't seem to converge. Can anybody give me some hints on how to tweak the options? If a 100MB tar ball is not too much for you, the matrix is here: www.phys.ethz.ch/~cmay/binaryoutput.tgz Thanks in advance Christian From hzhang at mcs.anl.gov Sun Aug 23 21:44:29 2009 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Sun, 23 Aug 2009 21:44:29 -0500 (CDT) Subject: hints wanted for huge eigenvalue problem In-Reply-To: References: Message-ID: Christian, > > I want to extract the lowest eigenvalues of a huge generalized eigenvalue > problem with very dense clusters of eigenvalues. > I use slepc and superlu as external direct solver with the following options > > -eps_ncv 300 -eps_nev 220 -eps_tol 1e-10 > -st_ksp_rtol 1e-14 -st_ksp_type preonly -st_pc_type lu > -st_pc_factor_mat_solver_package superlu_dist > > Without shift-and-invert, this simply takes way too long, even on 48 CPUs it > runs for hours. > > If I use a shift-and-invert technique additionally invoking > -st_type sinvert -st_shift -0.41, > > it converges very fast and also parallelizes well, but I most likely don't > get the lowest eigenvalues if st_shift is slightly too high. If it's slightly > too low, it doesn't seem to converge. Clustered eigenvalue problems are extremely difficult to solve. In most cases, the shift-and-invert technique is the only practical approach, for which the selection of the shift is also sensitive, as what you observed. One suggestion is to try larger -eps_ncv, e.g., eps_ncv = 2*eps_nev, which might improve the robustness. > > Can anybody give me some hints on how to tweak the options? > If a 100MB tar ball is not too much for you, the matrix is here: > www.phys.ethz.ch/~cmay/binaryoutput.tgz Are your matrices symmetric/Hermitian? You may send this request to slepc's developers who are more experienced than us on eigenvalue problems. Hong From knepley at gmail.com Mon Aug 24 01:24:32 2009 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 24 Aug 2009 10:24:32 +0400 Subject: linear elliptic vector equation In-Reply-To: References: <74e91d510908180626t2b2d44b4u93be7b9264b0bd36@mail.gmail.com> <194BBA12-40EF-4EB8-8FAF-552F69E87FDC@mcs.anl.gov> <935B6944-C0D3-4F60-A02E-3A72A597DF8A@mcs.anl.gov> Message-ID: On 8/20/09, nicolas aunai wrote: > > Hi, I will answer the questions that I can below: I've written a little code based on the exercice 29, to solve my > equation (u - mu*laplacian(u) = f) with one DOF and with dirichlet > boundary conditions. > > my code is here : > > http://nau.cetp.ipsl.fr/inertia1.c > > It seems that the code is *very* slow (never waited the end...) when > the dmmg_nlevels is > 4 (with a 3x3 coarsest grid). Send the output of -log_summary. The refinement factor has the default value 2, and I want my fine grid > to be for example nx=513, ny=257 grid points. > > > I have some more questions : > > 1/ How can I change the linear solver and preconditionner *in* the > code, not with the command -ksp_type and -pc_type ? You can use DMMGGetKSP() and then KSPGetPC(), and then extract the various parts of the MG PC. However, I would never ever do this. First, it precludes nice testing you can do by just setting the PC to LU and the KSP to preonly. Furthermore, it seems to preclude nice optimizations to be made later. I really really really recommend that you do not do this. 2/ What and Where is actually used the linear solver and the > preconditionners ? Is it used at each grid level as a smoother ? Can > we change the pre/post smoothing parameters ? Yes, and yes. You can always use -help to see all the options that can customize the solve. 3/ My code is written to work in parallel, can I use a direct solver > when I'm at the coarset grid (small matrix) ? Yes. The default is to solve the coarse problem redundantly using LU. You can also use a parallel 3rd party LU. 4/ When I add -ksp_monitor, do I see the residual of the smoothing > iterations at each grid level ? If no, what is it ? I believe you can use -dmmg_ksp_monitor, however I can't check right now since I am at Domodedovo airport. You can check using -help. 5/ My code is just a test code with some analytical RHS. In the end, > the solver is supposed to be added in a simulation code where the RHS > will be a/ the result of some other calculation, b/ changing with time > (the system being solved at each time step). Is it relevant to look at > the timing with this test code since the RHS is not the one that is > supposed to be (i.e. is the RHS very important for convergence speed) > ? This depends completely on the particular problem. Usually not. 6/ When the solver will be added in the simulation, will it be much > more efficient to setup the solution vector with the previous solution > as an initial guess ? Yes, usually if you are solving for the full solution and not an update. 7/ My goal is not to do full dirichlet conditions like I have done in > the test code above, but rather DA_XPERIODIC and dirichlet in the Y > direction. I have observed that with a refinement factor of 2, and a > coarsest grid (nx=3,ny=3), when dmmg_nlevels is set to 2 I don't have > a 5x5 grid but a 6x5 grid, why ? for dmmg_nlevels = 8 I have 384x257 > grid instead of the 257^2 grid I was expected... what is the reason > for this ? > > 8/ With DA_XPERIODIC, am I supposed to use the stencil as if I were > within the domain when I'm on periodic "borders", i.e. is col[1].i = > i-1 is ok when i=0 (left border) ? Yes. Matt Ok that's it for the moment, > Thx > > > Nico > > > > 2009/8/19 Barry Smith : > > > > On Aug 19, 2009, at 2:04 AM, nicolas aunai wrote: > > > >> Hi, > >> > >> OK then, lets go for multigrid. > >> > >> > >> When I'm looking for example at exercice 32 : > >> > >> > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/ksp/ksp/examples/tutorials/ex32.c.html > >> > >> I realize that I don't understand some things, in particular about the > >> DMMGSetKSP function. > >> > >> I need to give the RHS function and the ComputeMatrix function. > >> > >> 1/ My operator is always the same during the simulation, I was > >> thinking to build the matrix once and for all at the begining of the > >> program. Why can't I do that ? Is it because at each multigrid level > >> the required matrix is different (size) and it can only be built with > >> the information the DA is giving ? > > > > Yes. The idea is rather than you giving the matrix. You give the > function > > that fills up the matrix. > >> > >> 2/ I don't really understand the prototype of the callback function > >> ComputeMatrix. The first argument is the DMMG object, ok. But the > >> second and third arguments are matrices called 'J' and 'jac'. It > >> seems (at least in ex32) that 'J' is not used, what is the goal of > >> this matrix in general ? > > > > Sometimes one uses say a second order discretization to define the > system > > but wants a cheap preconditioner built from the first order > discretization. > > Hence the two matrices. If you are not doing this then you pass the same > > matrix in both slots. > > > >> > >> Is 'jac' stands for jacobian ? Reading the ComputeMatrix() code, it > >> seems that the 'jac' matrix is built like a standard finite difference > >> matrix, with the stencil corresponding to the operator I have to > >> solve. if so, why is it called 'jacobian' ? (sorry it looks more a > >> math question than a petsc question...) > > > > This is just brought over from the nonlinear solver code. It could just > be > > called matrix here. > > > > Barry > > > >> > >> > >> Another quick question : > >> > >> Why is there no documentation for the function 'DMMGGetr(dmmg)' which > >> seems to be part of the petsc library since I can't see the definition > >> of the function in the exercice file. I suppose it computes the > >> residual ? > >> > >> > >> I guess this is it.... for the moment. > >> Thx > >> > >> Nico > >> > >> > >> > >> 2009/8/19 Barry Smith : > >>> > >>> On Aug 18, 2009, at 11:42 AM, nicolas aunai wrote: > >>> > >>>> Hello, > >>>> > >>>> Yes it is a bit overkill indeed :-) I was looking at ex29 actually... > >>>> which is still linear and with DA's but scalar. > >>>> > >>>> But I'd rather like not use multigrids, however I have not seen an > >>>> example of a code that uses DA's and KSP solvers. > >>>> > >>>> Every example use multigrids. > >>> > >>> Multigrid is the way to go for this problem, it will be faster than > some > >>> generic preconditioned CG. > >>> > >>>> > >>>> When you have a multicomponent linear equation to solve on a > >>>> rectangular uniform grid such as mine, is it good to : > >>>> > >>>> 1/ use DA's to represent your grid > >>>> > >>>> 2/ create the matrix corresponding to your operator once and for all > >>>> (the operator neve changes).( In fact there are 2 matrices because two > >>>> cases possibles (Neumann/periodic or Dirichlet/Periodic, depending on > >>>> the component). but never minds) > >>>> > >>>> > >>>> 3/ Call KSPSolve() 3 times (for each vector component associated with > >>>> the > >>>> DA) ? > >>>> > >>> Yes. You can do all this using the DMMG object. > >>> > >>> Note that the DMMG object in your linear case just manages a KSP and > the > >>> meshes for you. In the end it is a KSPSolve() that actually does the > >>> solve > >>> and you have full control over the solve including using -pc_type icc > or > >>> something else that does not use multigrid. Though multigrid will be > >>> faster. > >>> > >>> > >>> Barry > >>> > >>> > >>>> > >>>> Thx > >>>> Nico > >>>> > >>>> > >>>> 2009/8/18 Aron Ahmadia : > >>>>> > >>>>> ex19 comes to mind, though it's a bit overkill for what you're > doing... > >>>>> > >>>>> > >>>>> > >>>>> > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/src/snes/examples/tutorials/ex19.c.html > >>>>> > >>>>> ex19 uses DAs and DMMG, which is kind of like a meta-DA for using > >>>>> multigrid-style solvers. Both work well with structured grids. > >>>>> > >>>>> You may also want to review the PETSc manual section on DA. > >>>>> > >>>>> A > >>>>> > >>>>> On Tue, Aug 18, 2009 at 3:30 AM, nicolas aunai > >>>>> > >>>>> wrote: > >>>>>> > >>>>>> Dear all, > >>>>>> > >>>>>> > >>>>>> I have a simulation code (particle in cell) in which I have to solve > a > >>>>>> linear vector equation 4 times per time step. A typical run consists > >>>>>> of 50 000 time steps. The grid is rectangular and uniform of size > Lx, > >>>>>> Ly, with nx+1 and ny+1 points in the x and y direction > respectively., > >>>>>> (nx, ny) could be at max (1024,1024). > >>>>>> > >>>>>> The vector equation is the following : > >>>>>> > >>>>>> B(x,y) - a*Laplacian(B(x,y)) = S(x,y) > >>>>>> > >>>>>> > >>>>>> Where B and S are 2D vector fields with 3 components (Bx, By, Bz and > >>>>>> Sx, Sy, Sz), each depending on the x and y coordinates. > >>>>>> > >>>>>> > >>>>>> 'a' is a positive constant, smaller than one, typically a= 0.02 > >>>>>> > >>>>>> Boundary conditions may depend on for which B component we are > solving > >>>>>> the equation. On the x=cst borders, the boundary is always periodic, > >>>>>> no matter what component is solved, but on y=cst borders, the > boundary > >>>>>> condition can be either Neumann or Dirichlet. > >>>>>> > >>>>>> So far, I have written a small code that creates a vector solution, > a > >>>>>> vector RHS and the matrix operator, and solve a scalar equation of > >>>>>> this type. Solving my vector equation would then just call 3 times > >>>>>> this kind of code... but I believe there is another way to deal with > >>>>>> vector fields and linear systems with Petsc, using DAs no ? Could > >>>>>> someone explain me how solve this the proper way and/or show me some > >>>>>> code solving linear system with vector fields ? (is there an example > >>>>>> in the exercices that I would not have seen ?) > >>>>>> > >>>>>> > >>>>>> > >>>>>> Thanks a lot > >>>>>> Nico > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Aron Jamil Ahmadia > >>>>> Assistant Research Scientist > >>>>> King Abdullah University of Science and Technology > >>>>> > >>> > >>> > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 24 14:38:05 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 24 Aug 2009 14:38:05 -0500 Subject: Matrix-free multigrid In-Reply-To: <4A8CC8DB.1000202@yale.edu> References: <4A8CC8DB.1000202@yale.edu> Message-ID: <17C6578F-1E5F-4C7C-8CA6-AEE47C3C8D36@mcs.anl.gov> Hal, The high-level DMMG interface to PETSc's linear multigrid currently only generates AIJ or BAIJ matrices and hence the easy support for -dmmg_grid_sequence only works for AIJ or BAIJ matrices. Sorry about this. A temporary fix for this would be to modify DAGetMatrix() to also accept MATMFFD type and generate the right matrix in that case. If you know C you can easily add this; the code is in src/dm/da/utils/fdda.c Barry My long term plan is for DAGetMatrix() to actually take as input a matrix whose type has been set and then just add additional information to it. For matrix-free it would just add the matrix sizes, for AIJ and BAIJ it would add the preallocation and nonzero structure. On Aug 19, 2009, at 10:54 PM, Hal Finkel wrote: > Hello, > > I'd like to try using a matrix-free Newton method with a multigrid > object configured with -dmmg_grid_sequence, but I don't understand > how. Using SNES directly, I can set -snes_mf, but this does not > seems to work with the multigrid object, and DMMGSetSNES() always > allocates a matrix. > > Thank you in advance, > Hal From tsjb00 at hotmail.com Mon Aug 24 16:19:09 2009 From: tsjb00 at hotmail.com (tsjb00) Date: Mon, 24 Aug 2009 21:19:09 +0000 Subject: fortran interface for MatCreateSeqAIJWithArrays Message-ID: Hi, there! I have a quick question on the Fortran interface of MatCreateSeqAIJWithArrays. The documentation indicates 'The i and j indices are 0 based'. Is it so when called in Fortran? Should the array be 0 based too? Any precautions to be taken to avoid segmentation error? Another question is about the storage of vector/arrays when using Fortran. In the PETSc examples, say a vector X_v is created using DACreateXXXVector, then the values are accessed with array X_a using DAVecGetArray. To access individual value, the loop is coded as: for (k=zs; k From bsmith at mcs.anl.gov Mon Aug 24 16:37:47 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 24 Aug 2009 16:37:47 -0500 Subject: fortran interface for MatCreateSeqAIJWithArrays In-Reply-To: References: Message-ID: On Aug 24, 2009, at 4:19 PM, tsjb00 wrote: > Hi, there! > > I have a quick question on the Fortran interface of > MatCreateSeqAIJWithArrays. > > The documentation indicates 'The i and j indices are 0 based'. This means the values in i and j start with zero; it does not mean you have to declare them as PetscInt i(0:a big number) you can declare them either as (0:a big number minus 1) or (1:a big number) > Is it so when called in Fortran? Should the array be 0 based too? That does not matter. > Any precautions to be taken to avoid segmentation error? This routine does NOT copy the values in the arrays. This means that the arrays that YOU pass in, i,j,a must exist until you finish using the matrix. If you allocate them then don't deallocate them until you destroy the matrix. If you declare them, for example PetscScalar a(somebignumber) then make sure this variable does not go out of scope until the matrix is destroyed. > > Another question is about the storage of vector/arrays when using > Fortran. In the PETSc examples, say a vector X_v is created using > DACreateXXXVector, then the values are accessed with array X_a using > DAVecGetArray. To access individual value, the loop is coded as: > for (k=zs; k for (j=ys; j for (i=xs; i for (idof=0;idof X_a[k][j][i][idof]=............. > ............................................... > &nb sp; } > } > } > } > If in Fortran, how should the loop be structured? Please check if > the following is right. > do k=zs,zs+zm > do j=ys,ys+ym > do i=xs,xs+xm > do iof=1,dof > X_a(iof,i,j,k)=.............. > enddo > ............. Yes. Barry > > > Many thanks in advance! > ????? Windows Live Messenger ???????? ????? From tsjb00 at hotmail.com Mon Aug 24 16:42:27 2009 From: tsjb00 at hotmail.com (tsjb00) Date: Mon, 24 Aug 2009 21:42:27 +0000 Subject: fortran interface for MatCreateSeqAIJWithArrays In-Reply-To: References: Message-ID: Many thanks for your reply! Just double check. So when I call MatCreateSeqAIJWithArrays from Fortran, I should define: i(1)=0 j(1)=0 Correct? Thanks again! > From: bsmith at mcs.anl.gov > To: petsc-users at mcs.anl.gov > Subject: Re: fortran interface for MatCreateSeqAIJWithArrays > Date: Mon, 24 Aug 2009 16:37:47 -0500 > > > On Aug 24, 2009, at 4:19 PM, tsjb00 wrote: > > > Hi, there! > > > > I have a quick question on the Fortran interface of > > MatCreateSeqAIJWithArrays. > > > > The documentation indicates 'The i and j indices are 0 based'. > > This means the values in i and j start with zero; it does not mean > you have to declare them as PetscInt i(0:a big number) you can declare > them either as > (0:a big number minus 1) or (1:a big number) > > > Is it so when called in Fortran? Should the array be 0 based too? > > That does not matter. > > > Any precautions to be taken to avoid segmentation error? > > This routine does NOT copy the values in the arrays. This means > that the arrays that YOU pass in, i,j,a must exist until you finish > using the matrix. If you allocate them > then don't deallocate them until you destroy the matrix. If you > declare them, for example PetscScalar a(somebignumber) then make sure > this variable does not go out of > scope until the matrix is destroyed. > > > > > Another question is about the storage of vector/arrays when using > > Fortran. In the PETSc examples, say a vector X_v is created using > > DACreateXXXVector, then the values are accessed with array X_a using > > DAVecGetArray. To access individual value, the loop is coded as: > > for (k=zs; k > for (j=ys; j > for (i=xs; i > for (idof=0;idof > X_a[k][j][i][idof]=............. > > ............................................... > > &nb sp; } > > } > > } > > } > > If in Fortran, how should the loop be structured? Please check if > > the following is right. > > do k=zs,zs+zm > > do j=ys,ys+ym > > do i=xs,xs+xm > > do iof=1,dof > > X_a(iof,i,j,k)=.............. > > enddo > > ............. > > Yes. > > Barry > > > > > > > Many thanks in advance! > > ????? Windows Live Messenger ???????? ????? > _________________________________________________________________ ?Windows Live ??????????Messenger? http://www.windowslive.cn -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Mon Aug 24 16:51:12 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 24 Aug 2009 16:51:12 -0500 Subject: fortran interface for MatCreateSeqAIJWithArrays In-Reply-To: References: Message-ID: <4CFE09F1-296D-4695-A543-7F937BB69F4A@mcs.anl.gov> Correct. On Aug 24, 2009, at 4:42 PM, tsjb00 wrote: > Many thanks for your reply! Just double check. So when I call > MatCreateSeqAIJWithArrays from Fortran, I should define: > i(1)=0 > j(1)=0 > Correct? > > Thanks again! > > > From: bsmith at mcs.anl.gov > > To: petsc-users at mcs.anl.gov > > Subject: Re: fortran interface for MatCreateSeqAIJWithArrays > > Date: Mon, 24 Aug 2009 16:37:47 -0500 > > > > > > On Aug 24, 2009, at 4:19 PM, tsjb00 wrote: > > > > > Hi, there! > > > > > > I have a quick question on the Fortran interface of > > > MatCreateSeqAIJWithArrays. > > > > > > The documentation indicates 'The i and j indices are 0 based'. > > > > This means the values in i and j start with zero; it does not mean > > you have to declare them as PetscInt i(0:a big number) you can > declare > > them either as > > (0:a big number minus 1) or (1:a big number) > > > > > Is it so when called in Fortran? Should the array be 0 based too? > > > > That does not matter. > > > > > Any precautions to be taken to avoid segmentation error? > > > > This routine does NOT copy the values in the arrays. This means > > that the arrays that YOU pass in, i,j,a must exist until you finish > > using the matrix. If you allocate them > > then don't deallocate them until you destroy the matrix. If you > > declare them, for example PetscScalar a(somebignumber) then make > sure > > this variable does not go out of > > scope until the matrix is destroyed. > > > > > > > > Another question is about the storage of vector/arrays when using > > > Fortran. In the PETSc examples, say a vector X_v is created using > > > DACreateXXXVector, then the values are accessed with array X_a > using > > > DAVecGetArray. To access individual value, the loop is coded as: > > > for (k=zs; k > ; > for (j=ys; j > > for (i=xs; i > > for (idof=0;idof > > X_a[k][j][i][idof]=............. > > > ............................................... > > > &nb sp; } > > > } > > > } > > > } > > > If in Fortran, how should the loop be structured? Please check if > > > the following is right. > > > do k=zs,zs+zm > > > do j=ys,ys+ym > > > do i=xs,xs+xm > > > do iof=1,dof > > > X_a(iof,i,j,k)=.............. > > > enddo > > > ............. > > > > Yes. > > > > Barry > > > > > > > > > > > Many thanks in advance! > > > ????? Windows Live Messenger ???????? ????? > > > > ????????????? ????????????? ???! ??? > ?? From thomas.witkowski at tu-dresden.de Tue Aug 25 05:47:12 2009 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Tue, 25 Aug 2009 12:47:12 +0200 Subject: Petsc with Intel MKL for blas/lapack Message-ID: <4A93C130.8000503@tu-dresden.de> Hi, I want to compile petsc on two larger HPC system, on both we use Intels MKL for blas and lapack. The first one is a pc farm, and I've successfully compiled petsc with the parameter --with-blas-lapack-dir=/work/licsoft/libraries/mkl/10.0.3.020/lib/em64t This works fine. On the other system, its an SGI Altix, I've tried to do the same. But it fails in configure, because petsc tries to link the test program with "-lblas". The same test program on the pc farm was linked with the correct library parameter "-lmkl". May question is, why this can happen and what I can do to avoid the problem? Thanks for answering! Thomas From knepley at gmail.com Tue Aug 25 07:30:00 2009 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 25 Aug 2009 07:30:00 -0500 Subject: Petsc with Intel MKL for blas/lapack In-Reply-To: <4A93C130.8000503@tu-dresden.de> References: <4A93C130.8000503@tu-dresden.de> Message-ID: The biggest reason that the BLAS/LAPACK configure changes so rapidly among patchlevels is that MKL is the worst library for maintenance. They change the content AND the names of the libraries, as well as the link order, in almost every version. I am beginning to really dislike the developers personally. I suspect the difference is the exact patchlevel of PETSc used on the different machines. However, as always with configure problems, we cannot tell you anything at all unless you send configure.log. Matt On Tue, Aug 25, 2009 at 5:47 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Hi, > > I want to compile petsc on two larger HPC system, on both we use Intels MKL > for blas and lapack. The first one is a pc farm, and I've successfully > compiled petsc with the parameter > > --with-blas-lapack-dir=/work/licsoft/libraries/mkl/10.0.3.020/lib/em64t > > This works fine. On the other system, its an SGI Altix, I've tried to do > the same. But it fails in configure, because petsc tries to link the test > program with "-lblas". The same test program on the pc farm was linked with > the correct library parameter "-lmkl". May question is, why this can happen > and what I can do to avoid the problem? Thanks for answering! > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Tue Aug 25 13:25:39 2009 From: john.fettig at gmail.com (John Fettig) Date: Tue, 25 Aug 2009 13:25:39 -0500 Subject: Matrix creation/destruction Message-ID: I'm trying to understand where matrices get created and destroyed, because I have a mismatch in the -log_summary of my code but I can't figure out where. Running ksp/ksp/examples/tutorials/ex2 with 4 processors, the log summary looks like: Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 9 1.0 1.9534e-03 1.5 1.06e+03 1.1 5.4e+01 5.6e+01 0.0e+00 3 22 82 94 0 4 22100100 0 2 MatSolve 9 1.0 1.7405e-05 1.1 8.10e+02 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 0 18 0 0 0 186 MatLUFactorNum 1 1.0 4.2915e-05 5.3 7.60e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 7 MatILUFactorSym 1 1.0 1.2922e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 5 0 0 0 0 7 0 MatGetRowIJ 1 1.0 9.5367e-07 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 6.6996e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 3 0 0 0 0 5 0 VecMDot 8 1.0 1.3049e-03 2.0 9.72e+02 1.0 0.0e+00 0.0e+00 8.0e+00 2 21 0 0 12 2 21 0 0 20 3 VecNorm 10 1.0 6.4931e-03 1.2 2.80e+02 1.0 0.0e+00 0.0e+00 1.0e+01 12 6 0 0 15 14 6 0 0 24 0 VecScale 9 1.0 2.5988e-05 1.3 1.26e+02 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 19 VecCopy 1 1.0 3.0994e-06 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 12 1.0 1.3113e-05 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 2 1.0 2.2888e-05 1.8 5.60e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 10 VecMAXPY 9 1.0 2.2411e-05 1.9 1.23e+03 1.0 0.0e+00 0.0e+00 0.0e+00 0 27 0 0 0 0 27 0 0 0 220 VecScatterBegin 9 1.0 1.9288e-04 1.8 0.00e+00 0.0 5.4e+01 5.6e+01 0.0e+00 0 0 82 94 0 0 0100100 0 0 VecScatterEnd 9 1.0 9.6941e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 VecNormalize 9 1.0 6.1085e-03 1.1 3.78e+02 1.0 0.0e+00 0.0e+00 9.0e+00 12 8 0 0 14 13 8 0 0 22 0 KSPGMRESOrthog 8 1.0 1.3480e-03 1.9 1.98e+03 1.0 0.0e+00 0.0e+00 8.0e+00 2 44 0 0 12 2 44 0 0 20 6 KSPSetup 2 1.0 2.0330e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 KSPSolve 1 1.0 1.8885e-02 1.0 4.44e+03 1.0 4.8e+01 5.6e+01 3.5e+01 38 96 73 84 53 42 96 89 89 85 1 PCSetUp 2 1.0 5.5480e-03 1.2 7.60e+01 1.0 0.0e+00 0.0e+00 7.0e+00 10 2 0 0 11 11 2 0 0 17 0 PCSetUpOnBlocks 1 1.0 6.2799e-04 1.9 7.60e+01 1.0 0.0e+00 0.0e+00 5.0e+00 1 2 0 0 8 1 2 0 0 12 0 PCApply 9 1.0 2.3961e-04 1.1 8.10e+02 1.0 0.0e+00 0.0e+00 9.0e+00 0 18 0 0 14 1 18 0 0 22 14 --- Event Stage 1: Assembly MatAssemblyBegin 1 1.0 3.8123e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 1 0 0 0 3 5 0 0 0 17 0 MatAssemblyEnd 1 1.0 5.5060e-03 1.1 0.00e+00 0.0 1.2e+01 1.6e+01 1.0e+01 11 0 18 6 15 94 0100100 83 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. --- Event Stage 0: Main Stage Matrix 4 4 11408 0 Vec 20 21 29264 0 Vec Scatter 0 1 868 0 Index Set 3 3 1704 0 Krylov Solver 2 2 18880 0 Preconditioner 2 2 1408 0 --- Event Stage 1: Assembly Vec 2 1 1304 0 Vec Scatter 1 0 0 0 Index Set 2 2 1052 0 ======================================================================================================================== Why are there 4 matrices created/destroyed? Where does this happen? I only see one matrix being created/destroyed in the source, so it must be happening someplace in the KSP/PC. Also, there is a mismatch in the Vec and Vec Scatter counts in each of the stages, even though it would seem that (e.g.) all the Vec Scatters should be contained to the assembly stage. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 25 13:38:46 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Aug 2009 13:38:46 -0500 Subject: Matrix creation/destruction In-Reply-To: References: Message-ID: <42DCB1EA-8DC9-4952-8991-1443ABDEA5C6@mcs.anl.gov> The ILU factorization requires a matrix that is created inside the KSP operations. Plus it counts sequential matrices that are contained inside the parallel matrices so the matrix numbers will often be hard to understand. The VecScatter is created in the MatAssembly and used in the matrix vector products and then destroyed when the matrix is destroy that is why its creation and destruction are in different states. The index sets in the assembly are used to create the vecscatter. The index sets in the main stage a probably related to the preconditioner building. Barry On Aug 25, 2009, at 1:25 PM, John Fettig wrote: > I'm trying to understand where matrices get created and destroyed, > because I have a mismatch in the -log_summary of my code but I can't > figure out where. Running ksp/ksp/examples/tutorials/ex2 with 4 > processors, the log summary looks like: > > > > Object Type Creations Destructions Memory Descendants' > Mem. > > --- Event Stage 0: Main Stage > > Matrix 4 4 11408 0 > Vec 20 21 29264 0 > Vec Scatter 0 1 868 0 > Index Set 3 3 1704 0 > Krylov Solver 2 2 18880 0 > Preconditioner 2 2 1408 0 > > --- Event Stage 1: Assembly > > Vec 2 1 1304 0 > Vec Scatter 1 0 0 0 > Index Set 2 2 1052 0 > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > ====================================================================== > > > Why are there 4 matrices created/destroyed? Where does this > happen? I only see one matrix being created/destroyed in the > source, so it must be happening someplace in the KSP/PC. Also, > there is a mismatch in the Vec and Vec Scatter counts in each of the > stages, even though it would seem that (e.g.) all the Vec Scatters > should be contained to the assembly stage. > > John > From john.fettig at gmail.com Tue Aug 25 13:42:18 2009 From: john.fettig at gmail.com (John Fettig) Date: Tue, 25 Aug 2009 13:42:18 -0500 Subject: Matrix creation/destruction In-Reply-To: <42DCB1EA-8DC9-4952-8991-1443ABDEA5C6@mcs.anl.gov> References: <42DCB1EA-8DC9-4952-8991-1443ABDEA5C6@mcs.anl.gov> Message-ID: Barry, Thanks for the quick reply, that definitely clears it up. John On Tue, Aug 25, 2009 at 1:38 PM, Barry Smith wrote: > > The ILU factorization requires a matrix that is created inside the KSP > operations. Plus it counts sequential matrices that are contained inside the > parallel matrices so the matrix numbers will often be hard to understand. > > The VecScatter is created in the MatAssembly and used in the matrix > vector products and then destroyed when the matrix is destroy that is why > its creation and destruction are in different states. > > The index sets in the assembly are used to create the vecscatter. The > index sets in the main stage a probably related to the preconditioner > building. > > Barry > > On Aug 25, 2009, at 1:25 PM, John Fettig wrote: > > I'm trying to understand where matrices get created and destroyed, because >> I have a mismatch in the -log_summary of my code but I can't figure out >> where. Running ksp/ksp/examples/tutorials/ex2 with 4 processors, the log >> summary looks like: >> >> >> >> Object Type Creations Destructions Memory Descendants' Mem. >> >> --- Event Stage 0: Main Stage >> >> Matrix 4 4 11408 0 >> Vec 20 21 29264 0 >> Vec Scatter 0 1 868 0 >> Index Set 3 3 1704 0 >> Krylov Solver 2 2 18880 0 >> Preconditioner 2 2 1408 0 >> >> --- Event Stage 1: Assembly >> >> Vec 2 1 1304 0 >> Vec Scatter 1 0 0 0 >> Index Set 2 2 1052 0 >> >> ======================================================================================================================== >> >> >> Why are there 4 matrices created/destroyed? Where does this happen? I >> only see one matrix being created/destroyed in the source, so it must be >> happening someplace in the KSP/PC. Also, there is a mismatch in the Vec and >> Vec Scatter counts in each of the stages, even though it would seem that >> (e.g.) all the Vec Scatters should be contained to the assembly stage. >> >> John >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Aug 25 14:21:52 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 25 Aug 2009 14:21:52 -0500 (CDT) Subject: Petsc with Intel MKL for blas/lapack In-Reply-To: References: <4A93C130.8000503@tu-dresden.de> Message-ID: Actually we now use a notation with MKL library names that appears to work consistantly across MKL versions. So there could be other issues. configure.log will clear things up [so send it over to us at petsc-maint at mcs.anl.gov] Satish On Tue, 25 Aug 2009, Matthew Knepley wrote: > The biggest reason that the BLAS/LAPACK configure changes so rapidly among > patchlevels > is that MKL is the worst library for maintenance. They change the content > AND the names > of the libraries, as well as the link order, in almost every version. I am > beginning to really > dislike the developers personally. > > I suspect the difference is the exact patchlevel of PETSc used on the > different machines. However, > as always with configure problems, we cannot tell you anything at all unless > you send configure.log. > > Matt > > On Tue, Aug 25, 2009 at 5:47 AM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > > > Hi, > > > > I want to compile petsc on two larger HPC system, on both we use Intels MKL > > for blas and lapack. The first one is a pc farm, and I've successfully > > compiled petsc with the parameter > > > > --with-blas-lapack-dir=/work/licsoft/libraries/mkl/10.0.3.020/lib/em64t > > > > This works fine. On the other system, its an SGI Altix, I've tried to do > > the same. But it fails in configure, because petsc tries to link the test > > program with "-lblas". The same test program on the pc farm was linked with > > the correct library parameter "-lmkl". May question is, why this can happen > > and what I can do to avoid the problem? Thanks for answering! > > > > Thomas > > > > > > From john.fettig at gmail.com Tue Aug 25 16:33:38 2009 From: john.fettig at gmail.com (John Fettig) Date: Tue, 25 Aug 2009 16:33:38 -0500 Subject: Matrix creation/destruction In-Reply-To: References: <42DCB1EA-8DC9-4952-8991-1443ABDEA5C6@mcs.anl.gov> Message-ID: Another similar question. Running src/snes/examples/tutorials/ex1 on 2 processors with -log_summary shows VecScatterBegin 91 1.0 1.2181e-03 1.3 0.00e+00 0.0 3.8e+01 8.0e+00 0.0e+00 4 0 49 40 0 4 0 49 40 0 0 VecScatterEnd 19 1.0 3.5119e-04 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 There are 91 Begins but only 19 Ends. The calls are matched in the driver, so why aren't they matched when it runs? John On Tue, Aug 25, 2009 at 1:42 PM, John Fettig wrote: > Barry, > > Thanks for the quick reply, that definitely clears it up. > > John > > > On Tue, Aug 25, 2009 at 1:38 PM, Barry Smith wrote: > >> >> The ILU factorization requires a matrix that is created inside the KSP >> operations. Plus it counts sequential matrices that are contained inside the >> parallel matrices so the matrix numbers will often be hard to understand. >> >> The VecScatter is created in the MatAssembly and used in the matrix >> vector products and then destroyed when the matrix is destroy that is why >> its creation and destruction are in different states. >> >> The index sets in the assembly are used to create the vecscatter. The >> index sets in the main stage a probably related to the preconditioner >> building. >> >> Barry >> >> On Aug 25, 2009, at 1:25 PM, John Fettig wrote: >> >> I'm trying to understand where matrices get created and destroyed, >>> because I have a mismatch in the -log_summary of my code but I can't figure >>> out where. Running ksp/ksp/examples/tutorials/ex2 with 4 processors, the >>> log summary looks like: >>> >>> >>> >>> Object Type Creations Destructions Memory Descendants' Mem. >>> >>> --- Event Stage 0: Main Stage >>> >>> Matrix 4 4 11408 0 >>> Vec 20 21 29264 0 >>> Vec Scatter 0 1 868 0 >>> Index Set 3 3 1704 0 >>> Krylov Solver 2 2 18880 0 >>> Preconditioner 2 2 1408 0 >>> >>> --- Event Stage 1: Assembly >>> >>> Vec 2 1 1304 0 >>> Vec Scatter 1 0 0 0 >>> Index Set 2 2 1052 0 >>> >>> ======================================================================================================================== >>> >>> >>> Why are there 4 matrices created/destroyed? Where does this happen? I >>> only see one matrix being created/destroyed in the source, so it must be >>> happening someplace in the KSP/PC. Also, there is a mismatch in the Vec and >>> Vec Scatter counts in each of the stages, even though it would seem that >>> (e.g.) all the Vec Scatters should be contained to the assembly stage. >>> >>> John >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Tue Aug 25 16:44:09 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 25 Aug 2009 16:44:09 -0500 Subject: Matrix creation/destruction In-Reply-To: References: <42DCB1EA-8DC9-4952-8991-1443ABDEA5C6@mcs.anl.gov> Message-ID: <773ECFE9-BB69-4619-8B47-2DB1E5B65DAD@mcs.anl.gov> Sequential scatters and purely local parallel scatters don't have an end that is logged because the begin does everything. They return immediately without logging the end, hence the confusion. Barry On Aug 25, 2009, at 4:33 PM, John Fettig wrote: > Another similar question. Running src/snes/examples/tutorials/ex1 > on 2 processors with -log_summary shows > > VecScatterBegin 91 1.0 1.2181e-03 1.3 0.00e+00 0.0 3.8e+01 8.0e > +00 0.0e+00 4 0 49 40 0 4 0 49 40 0 0 > VecScatterEnd 19 1.0 3.5119e-04 3.5 0.00e+00 0.0 0.0e+00 0.0e > +00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > > There are 91 Begins but only 19 Ends. The calls are matched in the > driver, so why aren't they matched when it runs? > > John > > On Tue, Aug 25, 2009 at 1:42 PM, John Fettig > wrote: > Barry, > > Thanks for the quick reply, that definitely clears it up. > > John > > > On Tue, Aug 25, 2009 at 1:38 PM, Barry Smith > wrote: > > The ILU factorization requires a matrix that is created inside the > KSP operations. Plus it counts sequential matrices that are > contained inside the parallel matrices so the matrix numbers will > often be hard to understand. > > The VecScatter is created in the MatAssembly and used in the > matrix vector products and then destroyed when the matrix is destroy > that is why its creation and destruction are in different states. > > The index sets in the assembly are used to create the vecscatter. > The index sets in the main stage a probably related to the > preconditioner building. > > Barry > > > On Aug 25, 2009, at 1:25 PM, John Fettig wrote: > > I'm trying to understand where matrices get created and destroyed, > because I have a mismatch in the -log_summary of my code but I can't > figure out where. Running ksp/ksp/examples/tutorials/ex2 with 4 > processors, the log summary looks like: > > > > Object Type Creations Destructions Memory Descendants' > Mem. > > --- Event Stage 0: Main Stage > > Matrix 4 4 11408 0 > Vec 20 21 29264 0 > Vec Scatter 0 1 868 0 > Index Set 3 3 1704 0 > Krylov Solver 2 2 18880 0 > Preconditioner 2 2 1408 0 > > --- Event Stage 1: Assembly > > Vec 2 1 1304 0 > Vec Scatter 1 0 0 0 > Index Set 2 2 1052 0 > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > = > ====================================================================== > > > Why are there 4 matrices created/destroyed? Where does this > happen? I only see one matrix being created/destroyed in the > source, so it must be happening someplace in the KSP/PC. Also, > there is a mismatch in the Vec and Vec Scatter counts in each of the > stages, even though it would seem that (e.g.) all the Vec Scatters > should be contained to the assembly stage. > > John > > > > From hal.finkel at yale.edu Wed Aug 26 14:34:20 2009 From: hal.finkel at yale.edu (Hal Finkel) Date: Wed, 26 Aug 2009 15:34:20 -0400 Subject: Multigrid Questions Message-ID: <4A958E3C.2070400@yale.edu> Hello, I've constructed a evolution scheme for a time-dependent 3D PDE system using the DMMG layer. The core of the code is essentially a loop around DMMGSolve in the spirit of snes/examples/tutorials/ex29.c. If I only run with one level, then the code runs and produces sensible results. Passing the option -mg_levels_0_ksp_type bcgsl increases the solver speed by about 30%, and -mg_levels_0_pc_type fieldsplit gives me another 5% improvement. Does that say something about the structure of my problem? If I run with two levels, the coarse solve seems to work, but the fine grid's KSP never seems to complete. If I run with -mg_levels_1_ksp_monitor_true_residual, I can see the residual does not seem to be converging at all: 0 KSP preconditioned resid norm 1.979915874387e+02 true resid norm 1.889855551272e+04 ||Ae||/||Ax|| 1.889855551272e+04 0 KSP preconditioned resid norm 3.833279709488e+06 true resid norm 3.665290817713e+08 ||Ae||/||Ax|| 3.665290817713e+08 0 KSP preconditioned resid norm 5.080465654647e+01 true resid norm 8.284057252719e+03 ||Ae||/||Ax|| 8.284057252719e+03 1 KSP preconditioned resid norm 6.057649668915e+00 true resid norm 5.136009463814e+02 ||Ae||/||Ax|| 5.136009463814e+02 0 KSP preconditioned resid norm 5.511785811223e+04 true resid norm 1.055162492334e+07 ||Ae||/||Ax|| 1.055162492334e+07 0 KSP preconditioned resid norm 2.164672340919e+01 true resid norm 2.999406329054e+04 ||Ae||/||Ax|| 2.999406329054e+04 0 KSP preconditioned resid norm 5.275983407134e+05 true resid norm 9.196351032040e+08 ||Ae||/||Ax|| 9.196351032040e+08 0 KSP preconditioned resid norm 3.338344860439e+01 true resid norm 5.133553722923e+03 ||Ae||/||Ax|| 5.133553722923e+03 1 KSP preconditioned resid norm 4.306407436023e+00 true resid norm 3.361745947223e+02 ||Ae||/||Ax|| 3.361745947223e+02 ... If I run with -pc_mg_monitor I see, after the course solve, only things like: Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 1.979915874387e+02 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 3.833279709488e+06 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 5.080465654647e+01 1 KSP Residual norm 6.057649668915e+00 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 5.511785811223e+04 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 2.164672340919e+01 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 5.275983407134e+05 Residual norms for mg_levels_1_ solve. 0 KSP Residual norm 3.338344860439e+01 1 KSP Residual norm 4.306407436023e+00 ... running with -info shows: [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver is diverging. Initial right hand size norm 0.0066162, current residual norm 23282.7 at iteration 0 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver is diverging. Initial right hand size norm 0.00287711, current residual norm 38.2205 at iteration 0 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver is diverging. Initial right hand size norm 0.00287711, current residual norm 809191 at iteration 0 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver is diverging. Initial right hand size norm 0.00970186, current residual norm 186.983 at iteration 0 [0] KSPDefaultConverged(): user has provided nonzero initial guess, computing 2-norm of preconditioned RHS [0] KSPDefaultConverged(): Linear solver is diverging. Initial right hand size norm 0.00970186, current residual norm 3.62292e+06 at iteration 0 What is a good way to debug this? No matter how I run, I've arranged the code so that the fine grid is *always* the same size; it has to be because the initial conditions are loaded from a file. In general, however, I don't understand exactly what the mg preconditioner is doing. Is there a writeup somewhere which is more verbose than the manual? In the one-level case, how are the results of the solve used to precondition the Jacobian? In an 1-processor, 1-level case, running with -snes_view gives: SNES Object: type: ls line search variant: SNESLineSearchCubic alpha=0.0001, maxstep=1e+08, minlambda=1e-12 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=36 total number of function evaluations=3 KSP Object: type: fgmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 right preconditioning PC Object: type: mg MG: type is FULL, levels=1 cycles=v Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- KSP Object:(mg_levels_0_) type: bcgsl BCGSL: Ell = 2 BCGSL: Delta = 0 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(mg_levels_0_) type: fieldsplit FieldSplit with MULTIPLICATIVE composition: total splits = 2, blocksize = 2 Solver info for each split is in the following KSP objects: Split number 0 Fields 0 KSP Object:(mg_levels_0_fieldsplit_0_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(mg_levels_0_fieldsplit_0_) type: ilu ILU: 0 levels of fill ILU: factor fill ratio allocated 1 ILU: tolerance for zero pivot 1e-12 ILU: using diagonal shift to prevent zero pivot ILU: using diagonal shift on blocks to prevent zero pivot out-of-place factorization matrix ordering: natural ILU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=19683, cols=19683 package used to perform factorization: petsc total: nonzeros=531441, allocated nonzeros=531441 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=19683, cols=19683 total: nonzeros=531441, allocated nonzeros=531441 not using I-node routines Split number 1 Fields 1 KSP Object:(mg_levels_0_fieldsplit_1_) type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(mg_levels_0_fieldsplit_1_) type: ilu ILU: 0 levels of fill ILU: factor fill ratio allocated 1 ILU: tolerance for zero pivot 1e-12 ILU: using diagonal shift to prevent zero pivot ILU: using diagonal shift on blocks to prevent zero pivot out-of-place factorization matrix ordering: natural ILU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=19683, cols=19683 package used to perform factorization: petsc total: nonzeros=531441, allocated nonzeros=531441 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=19683, cols=19683 total: nonzeros=531441, allocated nonzeros=531441 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=39366, cols=39366 total: nonzeros=2125764, allocated nonzeros=2125764 using I-node routines: found 19683 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=39366, cols=39366 total: nonzeros=2125764, allocated nonzeros=2125764 using I-node routines: found 19683 nodes, limit used is 5 Thank you in advance, Hal From knepley at gmail.com Wed Aug 26 14:41:38 2009 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 26 Aug 2009 14:41:38 -0500 Subject: Update on PETSc GPU In-Reply-To: <98KEoU018964@awe.co.uk> References: <98KEoU018964@awe.co.uk> Message-ID: On Thu, Aug 20, 2009 at 8:48 AM, Stephen Ball wrote: > Hi > > We are currently looking at procuring a GPU cluster for testing in the > near future. I would be very keen to test your PETSc GPU when it is > available. Can you give me an update on when you plan to make PETSc GPU > available? I don't think before the end of the year, because there is a lot to get done. I will post a repo soon, so you can always get the unfinished business just like petsc-dev. There is not a lot of PETSc that works well on a GPU, however, so I just want to list the main concerns: a) Sparse MV: Not great, and we plan to use the NVIDIA implementation b) Triangular Solve: Ditto, although we might rewrite some of it c) FMM: I would like to put in an interface to PetFMM-GPU d) Finite element integrals/stencils: There benefit greatly from GPU acceleration, but is not part of PETSc. Not sure how to do this, but we would work with users. e) Factorization: I hate this, but so many people do it. You probably only need the triangular solve part here, so it comes down to b) f) MG: This is jsut a bunch of a) and b) to organize d) is the most problematic, since its very important, but would demand a lot of user work I anticipate. Matt > Regards > > Stephen R. Ball > > This e-mail and any attachments may contain confidential and > privileged information. If you are not the intended recipient, > please notify the sender immediately by return e-mail, delete this > e-mail and destroy any copies. Any dissemination or use of this > information by a person other than the intended recipient is > unauthorized and may be illegal. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Aug 26 15:36:48 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Aug 2009 15:36:48 -0500 Subject: Multigrid Questions In-Reply-To: <4A958E3C.2070400@yale.edu> References: <4A958E3C.2070400@yale.edu> Message-ID: <2EF3551E-FEA7-4A09-BE94-41BA88E0190B@mcs.anl.gov> Please run with -ksp_monitor_true_residual -snes_monitor - mg_levels_ksp_monitor_true_residua -mg_coarse_ksp_type gmres - mg_coarse_ksp_monitor_true_residual -snes_view and send the output for one time-step. Barry On Aug 26, 2009, at 2:34 PM, Hal Finkel wrote: > Hello, > > I've constructed a evolution scheme for a time-dependent 3D PDE > system using the DMMG layer. The core of the code is essentially a > loop around DMMGSolve in the spirit of snes/examples/tutorials/ > ex29.c. If I only run with one level, then the code runs and > produces sensible results. Passing the option -mg_levels_0_ksp_type > bcgsl increases the solver speed by about 30%, and - > mg_levels_0_pc_type fieldsplit gives me another 5% improvement. Does > that say something about the structure of my problem? > > If I run with two levels, the coarse solve seems to work, but the > fine grid's KSP never seems to complete. If I run with - > mg_levels_1_ksp_monitor_true_residual, I can see the residual does > not seem to be converging at all: > > 0 KSP preconditioned resid norm 1.979915874387e+02 true resid > norm 1.889855551272e+04 ||Ae||/||Ax|| 1.889855551272e+04 > 0 KSP preconditioned resid norm 3.833279709488e+06 true resid > norm 3.665290817713e+08 ||Ae||/||Ax|| 3.665290817713e+08 > 0 KSP preconditioned resid norm 5.080465654647e+01 true resid > norm 8.284057252719e+03 ||Ae||/||Ax|| 8.284057252719e+03 > 1 KSP preconditioned resid norm 6.057649668915e+00 true resid > norm 5.136009463814e+02 ||Ae||/||Ax|| 5.136009463814e+02 > 0 KSP preconditioned resid norm 5.511785811223e+04 true resid > norm 1.055162492334e+07 ||Ae||/||Ax|| 1.055162492334e+07 > 0 KSP preconditioned resid norm 2.164672340919e+01 true resid > norm 2.999406329054e+04 ||Ae||/||Ax|| 2.999406329054e+04 > 0 KSP preconditioned resid norm 5.275983407134e+05 true resid > norm 9.196351032040e+08 ||Ae||/||Ax|| 9.196351032040e+08 > 0 KSP preconditioned resid norm 3.338344860439e+01 true resid > norm 5.133553722923e+03 ||Ae||/||Ax|| 5.133553722923e+03 > 1 KSP preconditioned resid norm 4.306407436023e+00 true resid > norm 3.361745947223e+02 ||Ae||/||Ax|| 3.361745947223e+02 > ... > > If I run with -pc_mg_monitor I see, after the course solve, only > things like: > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 1.979915874387e+02 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 3.833279709488e+06 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 5.080465654647e+01 > 1 KSP Residual norm 6.057649668915e+00 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 5.511785811223e+04 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 2.164672340919e+01 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 5.275983407134e+05 > Residual norms for mg_levels_1_ solve. > 0 KSP Residual norm 3.338344860439e+01 > 1 KSP Residual norm 4.306407436023e+00 > ... > > running with -info shows: > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > [0] KSPDefaultConverged(): Linear solver is diverging. Initial right > hand size norm 0.0066162, current residual norm 23282.7 at iteration 0 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > [0] KSPDefaultConverged(): Linear solver is diverging. Initial right > hand size norm 0.00287711, current residual norm 38.2205 at > iteration 0 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > [0] KSPDefaultConverged(): Linear solver is diverging. Initial right > hand size norm 0.00287711, current residual norm 809191 at iteration 0 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > [0] KSPDefaultConverged(): Linear solver is diverging. Initial right > hand size norm 0.00970186, current residual norm 186.983 at > iteration 0 > [0] KSPDefaultConverged(): user has provided nonzero initial guess, > computing 2-norm of preconditioned RHS > [0] KSPDefaultConverged(): Linear solver is diverging. Initial right > hand size norm 0.00970186, current residual norm 3.62292e+06 at > iteration 0 > > What is a good way to debug this? > > No matter how I run, I've arranged the code so that the fine grid is > *always* the same size; it has to be because the initial conditions > are loaded from a file. > > In general, however, I don't understand exactly what the mg > preconditioner is doing. Is there a writeup somewhere which is more > verbose than the manual? In the one-level case, how are the results > of the solve used to precondition the Jacobian? > > In an 1-processor, 1-level case, running with -snes_view gives: > SNES Object: > type: ls > line search variant: SNESLineSearchCubic > alpha=0.0001, maxstep=1e+08, minlambda=1e-12 > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=36 > total number of function evaluations=3 > KSP Object: > type: fgmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > right preconditioning > PC Object: > type: mg > MG: type is FULL, levels=1 cycles=v > Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- > KSP Object:(mg_levels_0_) > type: bcgsl > BCGSL: Ell = 2 > BCGSL: Delta = 0 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > PC Object:(mg_levels_0_) > type: fieldsplit > FieldSplit with MULTIPLICATIVE composition: total splits = > 2, blocksize = 2 > Solver info for each split is in the following KSP objects: > Split number 0 Fields 0 > KSP Object:(mg_levels_0_fieldsplit_0_) > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > PC Object:(mg_levels_0_fieldsplit_0_) > type: ilu > ILU: 0 levels of fill > ILU: factor fill ratio allocated 1 > ILU: tolerance for zero pivot 1e-12 > ILU: using diagonal shift to prevent zero pivot > ILU: using diagonal shift on blocks to prevent zero pivot > out-of-place factorization > matrix ordering: natural > ILU: factor fill ratio needed 1 > Factored matrix follows > Matrix Object: > type=seqaij, rows=19683, cols=19683 > package used to perform factorization: petsc > total: nonzeros=531441, allocated nonzeros=531441 > not using I-node routines > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=19683, cols=19683 > total: nonzeros=531441, allocated nonzeros=531441 > not using I-node routines > Split number 1 Fields 1 > KSP Object:(mg_levels_0_fieldsplit_1_) > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, > divergence=10000 > left preconditioning > PC Object:(mg_levels_0_fieldsplit_1_) > type: ilu > ILU: 0 levels of fill > ILU: factor fill ratio allocated 1 > ILU: tolerance for zero pivot 1e-12 > ILU: using diagonal shift to prevent zero pivot > ILU: using diagonal shift on blocks to prevent zero pivot > out-of-place factorization > matrix ordering: natural > ILU: factor fill ratio needed 1 > Factored matrix follows > Matrix Object: > type=seqaij, rows=19683, cols=19683 > package used to perform factorization: petsc > total: nonzeros=531441, allocated nonzeros=531441 > not using I-node routines > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=19683, cols=19683 > total: nonzeros=531441, allocated nonzeros=531441 > not using I-node routines > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=39366, cols=39366 > total: nonzeros=2125764, allocated nonzeros=2125764 > using I-node routines: found 19683 nodes, limit used is 5 > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=39366, cols=39366 > total: nonzeros=2125764, allocated nonzeros=2125764 > using I-node routines: found 19683 nodes, limit used is 5 > > Thank you in advance, > Hal From hal.finkel at yale.edu Wed Aug 26 15:45:59 2009 From: hal.finkel at yale.edu (Hal Finkel) Date: Wed, 26 Aug 2009 16:45:59 -0400 Subject: Multigrid Questions In-Reply-To: <2EF3551E-FEA7-4A09-BE94-41BA88E0190B@mcs.anl.gov> References: <4A958E3C.2070400@yale.edu> <2EF3551E-FEA7-4A09-BE94-41BA88E0190B@mcs.anl.gov> Message-ID: <4A959F07.9010506@yale.edu> Barry, Here is the output with those options in the (working) 1-level case on 1 cpu. Is that what you wanted? 0 SNES Function norm 6.643976542462e-01 0 KSP preconditioned resid norm 6.643976542462e-01 true resid norm 6.643976542462e-01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.438911631652e-03 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.438911631652e-03 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.141505633982e-04 true resid norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 1 KSP preconditioned resid norm 5.141505633982e-04 true resid norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 1 KSP preconditioned resid norm 4.790430472989e-02 true resid norm 4.790430472989e-02 ||Ae||/||Ax|| 7.210185710881e-02 0 KSP preconditioned resid norm 8.314136308720e-03 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.314136308720e-03 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.510029589596e-03 true resid norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 1 KSP preconditioned resid norm 4.510029589596e-03 true resid norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 2 KSP preconditioned resid norm 8.487458086567e-03 true resid norm 8.487458086567e-03 ||Ae||/||Ax|| 1.277466594339e-02 0 KSP preconditioned resid norm 6.119493660032e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.119493660032e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.236002849750e-02 true resid norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 1 KSP preconditioned resid norm 2.236002849750e-02 true resid norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 3 KSP preconditioned resid norm 5.965466932886e-03 true resid norm 5.965466932886e-03 ||Ae||/||Ax|| 8.978759775506e-03 0 KSP preconditioned resid norm 1.765708232662e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.765708232662e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.422901024605e-02 true resid norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 1 KSP preconditioned resid norm 2.422901024605e-02 true resid norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 4 KSP preconditioned resid norm 4.622977909431e-03 true resid norm 4.622977909431e-03 ||Ae||/||Ax|| 6.958149054087e-03 0 KSP preconditioned resid norm 8.428635776507e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.428635776507e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.071459298111e-02 true resid norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 1 KSP preconditioned resid norm 1.071459298111e-02 true resid norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 5 KSP preconditioned resid norm 2.949006884559e-03 true resid norm 2.949006884560e-03 ||Ae||/||Ax|| 4.438617243322e-03 0 KSP preconditioned resid norm 4.221520918573e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.221520918573e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.795668593850e-02 true resid norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 1 KSP preconditioned resid norm 1.795668593850e-02 true resid norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 6 KSP preconditioned resid norm 1.327402359192e-03 true resid norm 1.327402359192e-03 ||Ae||/||Ax|| 1.997903440370e-03 0 KSP preconditioned resid norm 3.239330680409e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.239330680409e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.103583195381e-02 true resid norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 1 KSP preconditioned resid norm 1.103583195381e-02 true resid norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 7 KSP preconditioned resid norm 5.579319891254e-04 true resid norm 5.579319891255e-04 ||Ae||/||Ax|| 8.397561092512e-04 0 KSP preconditioned resid norm 2.166256693843e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.166256693843e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.527142494434e-02 true resid norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 1 KSP preconditioned resid norm 2.527142494434e-02 true resid norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 8 KSP preconditioned resid norm 1.841437265702e-04 true resid norm 1.841437265703e-04 ||Ae||/||Ax|| 2.771589053535e-04 0 KSP preconditioned resid norm 1.097958713832e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.097958713832e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.070826208815e-02 true resid norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 1 KSP preconditioned resid norm 4.070826208815e-02 true resid norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 9 KSP preconditioned resid norm 6.545738966234e-05 true resid norm 6.545738966232e-05 ||Ae||/||Ax|| 9.852140392726e-05 0 KSP preconditioned resid norm 9.668719808206e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.668719808206e-02 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.944112488019e-02 true resid norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 1 KSP preconditioned resid norm 7.944112488019e-02 true resid norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 10 KSP preconditioned resid norm 4.125461342831e-05 true resid norm 4.125461342821e-05 ||Ae||/||Ax|| 6.209325569492e-05 0 KSP preconditioned resid norm 3.008151203349e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.008151203349e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.247738676584e-02 true resid norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 1 KSP preconditioned resid norm 7.247738676584e-02 true resid norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 11 KSP preconditioned resid norm 3.280778128691e-05 true resid norm 3.280778128689e-05 ||Ae||/||Ax|| 4.937973678446e-05 0 KSP preconditioned resid norm 9.091470733399e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.091470733399e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.215403990109e-01 true resid norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 1 KSP preconditioned resid norm 1.215403990109e-01 true resid norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 12 KSP preconditioned resid norm 2.968550831459e-05 true resid norm 2.968550831435e-05 ||Ae||/||Ax|| 4.468033281669e-05 0 KSP preconditioned resid norm 1.024246289230e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.024246289230e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.302673603327e-02 true resid norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 1 KSP preconditioned resid norm 9.302673603327e-02 true resid norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 13 KSP preconditioned resid norm 2.521514028222e-05 true resid norm 2.521514028196e-05 ||Ae||/||Ax|| 3.795188035479e-05 0 KSP preconditioned resid norm 5.990912160630e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.990912160630e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.781990197263e-02 true resid norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 1 KSP preconditioned resid norm 9.781990197263e-02 true resid norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 14 KSP preconditioned resid norm 1.978375072845e-05 true resid norm 1.978375072804e-05 ||Ae||/||Ax|| 2.977697257298e-05 0 KSP preconditioned resid norm 1.008331187053e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.008331187053e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.294954965258e-01 true resid norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 1 KSP preconditioned resid norm 2.294954965258e-01 true resid norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 15 KSP preconditioned resid norm 1.488645974384e-05 true resid norm 1.488645974344e-05 ||Ae||/||Ax|| 2.240594867893e-05 0 KSP preconditioned resid norm 1.849227047229e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.849227047229e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.919984907088e-01 true resid norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 1 KSP preconditioned resid norm 2.919984907088e-01 true resid norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 16 KSP preconditioned resid norm 7.094849419546e-06 true resid norm 7.094849419588e-06 ||Ae||/||Ax|| 1.067861900813e-05 0 KSP preconditioned resid norm 1.388766130430e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.388766130430e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.878119571370e-01 true resid norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 1 KSP preconditioned resid norm 3.878119571370e-01 true resid norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 17 KSP preconditioned resid norm 4.097305136804e-06 true resid norm 4.097305136552e-06 ||Ae||/||Ax|| 6.166947023919e-06 1 SNES Function norm 5.850920566283e-06 0 KSP preconditioned resid norm 5.850920566283e-06 true resid norm 5.850920566283e-06 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.276021960717e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.276021960717e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.809774037292e-01 true resid norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 1 KSP preconditioned resid norm 3.809774037292e-01 true resid norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 1 KSP preconditioned resid norm 3.947856942305e-06 true resid norm 3.947856942305e-06 ||Ae||/||Ax|| 6.747411621096e-01 0 KSP preconditioned resid norm 2.497905632896e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.497905632896e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.608088417629e-01 true resid norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 1 KSP preconditioned resid norm 5.608088417629e-01 true resid norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 2 KSP preconditioned resid norm 3.305771768709e-06 true resid norm 3.305771768709e-06 ||Ae||/||Ax|| 5.650002817948e-01 0 KSP preconditioned resid norm 7.301425383804e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.301425383804e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.476223530528e-01 true resid norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 1 KSP preconditioned resid norm 3.476223530528e-01 true resid norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 3 KSP preconditioned resid norm 3.007329187251e-06 true resid norm 3.007329187251e-06 ||Ae||/||Ax|| 5.139924825815e-01 0 KSP preconditioned resid norm 2.880397337024e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.880397337024e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.471977831965e-01 true resid norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 1 KSP preconditioned resid norm 4.471977831965e-01 true resid norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 4 KSP preconditioned resid norm 3.001204159469e-06 true resid norm 3.001204159469e-06 ||Ae||/||Ax|| 5.129456340194e-01 0 KSP preconditioned resid norm 4.619989941290e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.619989941290e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.243648576810e-01 true resid norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 1 KSP preconditioned resid norm 3.243648576810e-01 true resid norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 5 KSP preconditioned resid norm 3.001115709599e-06 true resid norm 3.001115709599e-06 ||Ae||/||Ax|| 5.129305167624e-01 0 KSP preconditioned resid norm 4.829994865746e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.829994865746e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.325734548745e-01 true resid norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 1 KSP preconditioned resid norm 4.325734548745e-01 true resid norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 6 KSP preconditioned resid norm 2.974277803753e-06 true resid norm 2.974277803753e-06 ||Ae||/||Ax|| 5.083435623606e-01 0 KSP preconditioned resid norm 1.381129102428e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.381129102428e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.098881160663e-01 true resid norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 1 KSP preconditioned resid norm 3.098881160663e-01 true resid norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 7 KSP preconditioned resid norm 2.816838388951e-06 true resid norm 2.816838388951e-06 ||Ae||/||Ax|| 4.814350762483e-01 0 KSP preconditioned resid norm 1.559573417426e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.559573417426e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.532723445272e-01 true resid norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 1 KSP preconditioned resid norm 2.532723445272e-01 true resid norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 8 KSP preconditioned resid norm 2.572324455447e-06 true resid norm 2.572324455447e-06 ||Ae||/||Ax|| 4.396443989123e-01 0 KSP preconditioned resid norm 5.946831260059e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.946831260059e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.929968207615e-01 true resid norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 1 KSP preconditioned resid norm 3.929968207615e-01 true resid norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 9 KSP preconditioned resid norm 2.177455315353e-06 true resid norm 2.177455315353e-06 ||Ae||/||Ax|| 3.721560206954e-01 0 KSP preconditioned resid norm 5.315723299063e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.315723299063e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.628504511585e-01 true resid norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 1 KSP preconditioned resid norm 3.628504511585e-01 true resid norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 10 KSP preconditioned resid norm 1.409067985559e-06 true resid norm 1.409067985559e-06 ||Ae||/||Ax|| 2.408284251334e-01 0 KSP preconditioned resid norm 1.522258580690e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.522258580690e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.414238177111e-01 true resid norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 1 KSP preconditioned resid norm 2.414238177111e-01 true resid norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 11 KSP preconditioned resid norm 1.001503517728e-06 true resid norm 1.001503517728e-06 ||Ae||/||Ax|| 1.711702468667e-01 0 KSP preconditioned resid norm 8.244553153983e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.244553153983e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.244218748685e-01 true resid norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 1 KSP preconditioned resid norm 2.244218748685e-01 true resid norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 12 KSP preconditioned resid norm 7.556215980081e-07 true resid norm 7.556215980082e-07 ||Ae||/||Ax|| 1.291457625254e-01 0 KSP preconditioned resid norm 1.857974243745e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.857974243745e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.451523414794e-01 true resid norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 1 KSP preconditioned resid norm 3.451523414794e-01 true resid norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 13 KSP preconditioned resid norm 5.872140840954e-07 true resid norm 5.872140840955e-07 ||Ae||/||Ax|| 1.003626826656e-01 0 KSP preconditioned resid norm 2.611648570586e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.611648570586e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.740043981340e-01 true resid norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 1 KSP preconditioned resid norm 2.740043981340e-01 true resid norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 14 KSP preconditioned resid norm 4.352350316840e-07 true resid norm 4.352350316841e-07 ||Ae||/||Ax|| 7.438744497613e-02 0 KSP preconditioned resid norm 2.003016340281e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.003016340281e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.073409352245e-01 true resid norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 1 KSP preconditioned resid norm 4.073409352245e-01 true resid norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 15 KSP preconditioned resid norm 3.052867274622e-07 true resid norm 3.052867274624e-07 ||Ae||/||Ax|| 5.217755462648e-02 0 KSP preconditioned resid norm 1.286639096168e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.286639096168e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.656621503377e-01 true resid norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 1 KSP preconditioned resid norm 3.656621503377e-01 true resid norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 16 KSP preconditioned resid norm 1.955997188783e-07 true resid norm 1.955997188782e-07 ||Ae||/||Ax|| 3.343058868468e-02 0 KSP preconditioned resid norm 1.335903581256e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.335903581256e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.001895431683e-01 true resid norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 1 KSP preconditioned resid norm 3.001895431683e-01 true resid norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 17 KSP preconditioned resid norm 1.567782688997e-07 true resid norm 1.567782688996e-07 ||Ae||/||Ax|| 2.679548750039e-02 0 KSP preconditioned resid norm 1.651696143482e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.651696143482e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.713972348795e-01 true resid norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 1 KSP preconditioned resid norm 2.713972348795e-01 true resid norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 18 KSP preconditioned resid norm 1.010718356377e-07 true resid norm 1.010718356378e-07 ||Ae||/||Ax|| 1.727451851256e-02 0 KSP preconditioned resid norm 1.011081720758e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.011081720758e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.527570671391e-01 true resid norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 1 KSP preconditioned resid norm 5.527570671391e-01 true resid norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 19 KSP preconditioned resid norm 8.997938714701e-08 true resid norm 8.997938714709e-08 ||Ae||/||Ax|| 1.537867180519e-02 0 KSP preconditioned resid norm 1.537108246495e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.537108246495e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.298772065070e-01 true resid norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 1 KSP preconditioned resid norm 4.298772065070e-01 true resid norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 20 KSP preconditioned resid norm 8.711914509357e-08 true resid norm 8.711914509361e-08 ||Ae||/||Ax|| 1.488981846646e-02 0 KSP preconditioned resid norm 1.923097362850e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.923097362850e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.075948671586e-01 true resid norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 1 KSP preconditioned resid norm 3.075948671586e-01 true resid norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 21 KSP preconditioned resid norm 8.323067409578e-08 true resid norm 8.323067409577e-08 ||Ae||/||Ax|| 1.422522715065e-02 0 KSP preconditioned resid norm 9.819276869885e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.819276869885e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.279383354509e-01 true resid norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 1 KSP preconditioned resid norm 4.279383354509e-01 true resid norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 22 KSP preconditioned resid norm 7.909034553981e-08 true resid norm 7.909034553913e-08 ||Ae||/||Ax|| 1.351759003445e-02 0 KSP preconditioned resid norm 9.582482498733e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.582482498733e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.872542232901e-01 true resid norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 1 KSP preconditioned resid norm 4.872542232901e-01 true resid norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 23 KSP preconditioned resid norm 7.583943619887e-08 true resid norm 7.583943619831e-08 ||Ae||/||Ax|| 1.296196646992e-02 0 KSP preconditioned resid norm 1.129055084162e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.129055084162e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.940290407027e-01 true resid norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 1 KSP preconditioned resid norm 3.940290407027e-01 true resid norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 24 KSP preconditioned resid norm 7.198974227007e-08 true resid norm 7.198974226927e-08 ||Ae||/||Ax|| 1.230400266996e-02 0 KSP preconditioned resid norm 7.922461164088e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.922461164088e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.568631157402e-01 true resid norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 1 KSP preconditioned resid norm 3.568631157402e-01 true resid norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 25 KSP preconditioned resid norm 6.969681693595e-08 true resid norm 6.969681693509e-08 ||Ae||/||Ax|| 1.191211129010e-02 0 KSP preconditioned resid norm 8.506106408028e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.506106408028e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.674832821714e-01 true resid norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 1 KSP preconditioned resid norm 2.674832821714e-01 true resid norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 26 KSP preconditioned resid norm 6.713093474885e-08 true resid norm 6.713093474791e-08 ||Ae||/||Ax|| 1.147356796036e-02 0 KSP preconditioned resid norm 7.107668950850e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.107668950850e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.080542740675e-01 true resid norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 1 KSP preconditioned resid norm 3.080542740675e-01 true resid norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 27 KSP preconditioned resid norm 6.316951623357e-08 true resid norm 6.316951623251e-08 ||Ae||/||Ax|| 1.079650894537e-02 0 KSP preconditioned resid norm 9.816834307897e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.816834307897e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.035636714372e-01 true resid norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 1 KSP preconditioned resid norm 3.035636714372e-01 true resid norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 28 KSP preconditioned resid norm 5.685161562141e-08 true resid norm 5.685161562024e-08 ||Ae||/||Ax|| 9.716695856008e-03 0 KSP preconditioned resid norm 1.017862349675e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.017862349675e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.438638918414e-01 true resid norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 1 KSP preconditioned resid norm 3.438638918414e-01 true resid norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 29 KSP preconditioned resid norm 5.170566864610e-08 true resid norm 5.170566864501e-08 ||Ae||/||Ax|| 8.837185201757e-03 0 KSP preconditioned resid norm 8.897697988733e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.897697988733e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.213181899588e-01 true resid norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 1 KSP preconditioned resid norm 3.213181899588e-01 true resid norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 30 KSP preconditioned resid norm 4.885061460285e-08 true resid norm 4.885061460285e-08 ||Ae||/||Ax|| 8.349218563034e-03 0 KSP preconditioned resid norm 7.100457326024e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.100457326024e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.955812925302e-01 true resid norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 1 KSP preconditioned resid norm 6.955812925302e-01 true resid norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 31 KSP preconditioned resid norm 4.725291213013e-08 true resid norm 4.725291213007e-08 ||Ae||/||Ax|| 8.076149999775e-03 0 KSP preconditioned resid norm 4.185203199010e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.185203199010e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.196403938530e-01 true resid norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 1 KSP preconditioned resid norm 2.196403938530e-01 true resid norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 32 KSP preconditioned resid norm 4.547823851359e-08 true resid norm 4.547823851362e-08 ||Ae||/||Ax|| 7.772834718641e-03 0 KSP preconditioned resid norm 6.581109982585e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.581109982585e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.128181724937e-01 true resid norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 1 KSP preconditioned resid norm 3.128181724937e-01 true resid norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 33 KSP preconditioned resid norm 4.468618222477e-08 true resid norm 4.468618222483e-08 ||Ae||/||Ax|| 7.637461783764e-03 0 KSP preconditioned resid norm 8.120341282592e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.120341282592e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.218250119940e-01 true resid norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 1 KSP preconditioned resid norm 2.218250119940e-01 true resid norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 34 KSP preconditioned resid norm 4.314354409778e-08 true resid norm 4.314354409782e-08 ||Ae||/||Ax|| 7.373804448217e-03 0 KSP preconditioned resid norm 5.778947005677e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.778947005677e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.877938034235e-01 true resid norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 1 KSP preconditioned resid norm 3.877938034235e-01 true resid norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 35 KSP preconditioned resid norm 4.111383309670e-08 true resid norm 4.111383309672e-08 ||Ae||/||Ax|| 7.026899892239e-03 0 KSP preconditioned resid norm 7.589832929724e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.589832929724e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.735429451288e-01 true resid norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 1 KSP preconditioned resid norm 2.735429451288e-01 true resid norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 36 KSP preconditioned resid norm 3.896413302095e-08 true resid norm 3.896413302095e-08 ||Ae||/||Ax|| 6.659487610460e-03 0 KSP preconditioned resid norm 9.173185113175e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.173185113175e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.052187383165e-01 true resid norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 1 KSP preconditioned resid norm 3.052187383165e-01 true resid norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 37 KSP preconditioned resid norm 3.770537943429e-08 true resid norm 3.770537943431e-08 ||Ae||/||Ax|| 6.444349911635e-03 0 KSP preconditioned resid norm 1.137492649970e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.137492649970e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.044329763751e-01 true resid norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 1 KSP preconditioned resid norm 3.044329763751e-01 true resid norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 38 KSP preconditioned resid norm 3.694716763304e-08 true resid norm 3.694716763306e-08 ||Ae||/||Ax|| 6.314761448988e-03 0 KSP preconditioned resid norm 1.223189932062e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.223189932062e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.107514686273e-01 true resid norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 1 KSP preconditioned resid norm 2.107514686273e-01 true resid norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 39 KSP preconditioned resid norm 3.634742486594e-08 true resid norm 3.634742486593e-08 ||Ae||/||Ax|| 6.212257448064e-03 0 KSP preconditioned resid norm 1.192363148419e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.192363148419e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.860271918718e-01 true resid norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 1 KSP preconditioned resid norm 2.860271918718e-01 true resid norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 40 KSP preconditioned resid norm 3.561782722372e-08 true resid norm 3.561782722374e-08 ||Ae||/||Ax|| 6.087559525075e-03 0 KSP preconditioned resid norm 8.881324685300e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.881324685300e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.642271412468e-01 true resid norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 1 KSP preconditioned resid norm 2.642271412468e-01 true resid norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 41 KSP preconditioned resid norm 3.506823410956e-08 true resid norm 3.506823410956e-08 ||Ae||/||Ax|| 5.993626765614e-03 0 KSP preconditioned resid norm 9.513827527161e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.513827527161e-01 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.263980500382e-01 true resid norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 1 KSP preconditioned resid norm 2.263980500382e-01 true resid norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 42 KSP preconditioned resid norm 3.466236527477e-08 true resid norm 3.466236527477e-08 ||Ae||/||Ax|| 5.924258393546e-03 0 KSP preconditioned resid norm 1.407625976444e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.407625976444e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.841824427477e-01 true resid norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 1 KSP preconditioned resid norm 1.841824427477e-01 true resid norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 43 KSP preconditioned resid norm 3.394294635279e-08 true resid norm 3.394294635284e-08 ||Ae||/||Ax|| 5.801300148978e-03 0 KSP preconditioned resid norm 1.536263909707e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.536263909707e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.972519671057e-01 true resid norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 1 KSP preconditioned resid norm 2.972519671057e-01 true resid norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 44 KSP preconditioned resid norm 3.248296724023e-08 true resid norm 3.248296724025e-08 ||Ae||/||Ax|| 5.551770336352e-03 0 KSP preconditioned resid norm 1.783804105384e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.783804105384e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.722881763352e-01 true resid norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 1 KSP preconditioned resid norm 3.722881763352e-01 true resid norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 45 KSP preconditioned resid norm 2.950320556254e-08 true resid norm 2.950320556258e-08 ||Ae||/||Ax|| 5.042489507138e-03 0 KSP preconditioned resid norm 2.201496546447e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.201496546447e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.087649817784e-01 true resid norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 1 KSP preconditioned resid norm 4.087649817784e-01 true resid norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 46 KSP preconditioned resid norm 2.319584580786e-08 true resid norm 2.319584580791e-08 ||Ae||/||Ax|| 3.964477990280e-03 0 KSP preconditioned resid norm 1.736355434315e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.736355434315e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.867386399797e-01 true resid norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 1 KSP preconditioned resid norm 6.867386399797e-01 true resid norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 47 KSP preconditioned resid norm 2.008430277340e-08 true resid norm 2.008430277343e-08 ||Ae||/||Ax|| 3.432673977693e-03 0 KSP preconditioned resid norm 1.488127968937e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.488127968937e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.516786708506e-01 true resid norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 1 KSP preconditioned resid norm 5.516786708506e-01 true resid norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 48 KSP preconditioned resid norm 1.854206408497e-08 true resid norm 1.854206408497e-08 ||Ae||/||Ax|| 3.169084911497e-03 0 KSP preconditioned resid norm 1.380328436388e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.380328436388e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.575661767087e-01 true resid norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 1 KSP preconditioned resid norm 4.575661767087e-01 true resid norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 49 KSP preconditioned resid norm 1.665547510085e-08 true resid norm 1.665547510085e-08 ||Ae||/||Ax|| 2.846641808270e-03 0 KSP preconditioned resid norm 2.313536103290e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.313536103290e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.305138630648e-01 true resid norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 1 KSP preconditioned resid norm 5.305138630648e-01 true resid norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 50 KSP preconditioned resid norm 1.462283172240e-08 true resid norm 1.462283172247e-08 ||Ae||/||Ax|| 2.499236070088e-03 0 KSP preconditioned resid norm 3.213123410447e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.213123410447e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.224884980518e-01 true resid norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 1 KSP preconditioned resid norm 5.224884980518e-01 true resid norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 51 KSP preconditioned resid norm 1.223657774649e-08 true resid norm 1.223657774655e-08 ||Ae||/||Ax|| 2.091393586347e-03 0 KSP preconditioned resid norm 2.023552916685e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.023552916685e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.574715128775e-01 true resid norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 1 KSP preconditioned resid norm 5.574715128775e-01 true resid norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 52 KSP preconditioned resid norm 9.853979905043e-09 true resid norm 9.853979905046e-09 ||Ae||/||Ax|| 1.684175984516e-03 0 KSP preconditioned resid norm 1.170745734096e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.170745734096e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.134603323396e-01 true resid norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 1 KSP preconditioned resid norm 4.134603323396e-01 true resid norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 53 KSP preconditioned resid norm 8.181596904133e-09 true resid norm 8.181596904190e-09 ||Ae||/||Ax|| 1.398343527570e-03 0 KSP preconditioned resid norm 2.385292955090e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.385292955090e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.286979495932e-01 true resid norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 1 KSP preconditioned resid norm 5.286979495932e-01 true resid norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 54 KSP preconditioned resid norm 6.409203469073e-09 true resid norm 6.409203469062e-09 ||Ae||/||Ax|| 1.095417959696e-03 0 KSP preconditioned resid norm 5.707419973376e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.707419973376e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.431212555078e-01 true resid norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 1 KSP preconditioned resid norm 6.431212555078e-01 true resid norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 55 KSP preconditioned resid norm 4.450638029919e-09 true resid norm 4.450638029969e-09 ||Ae||/||Ax|| 7.606731247757e-04 0 KSP preconditioned resid norm 7.199725849178e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.199725849178e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.820447922491e-01 true resid norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 1 KSP preconditioned resid norm 8.820447922491e-01 true resid norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 56 KSP preconditioned resid norm 2.820717192827e-09 true resid norm 2.820717192685e-09 ||Ae||/||Ax|| 4.820980153004e-04 0 KSP preconditioned resid norm 3.954316557881e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.954316557881e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.174082394688e+00 true resid norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 1 KSP preconditioned resid norm 1.174082394688e+00 true resid norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 57 KSP preconditioned resid norm 1.751635661806e-09 true resid norm 1.751635661897e-09 ||Ae||/||Ax|| 2.993777888545e-04 0 KSP preconditioned resid norm 1.876208896020e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.876208896020e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.566896137837e+00 true resid norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 1 KSP preconditioned resid norm 1.566896137837e+00 true resid norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 58 KSP preconditioned resid norm 1.207891841472e-09 true resid norm 1.207891841440e-09 ||Ae||/||Ax|| 2.064447513440e-04 0 KSP preconditioned resid norm 2.297261201903e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.297261201903e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.961454513785e+00 true resid norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 1 KSP preconditioned resid norm 1.961454513785e+00 true resid norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 59 KSP preconditioned resid norm 1.016539279879e-09 true resid norm 1.016539279925e-09 ||Ae||/||Ax|| 1.737400582368e-04 0 KSP preconditioned resid norm 3.254668263088e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.254668263088e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.640547979130e+00 true resid norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 1 KSP preconditioned resid norm 1.640547979130e+00 true resid norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 60 KSP preconditioned resid norm 8.558459393246e-10 true resid norm 8.558459393246e-10 ||Ae||/||Ax|| 1.462754330073e-04 0 KSP preconditioned resid norm 3.699188433787e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.699188433787e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.641405579973e+00 true resid norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 1 KSP preconditioned resid norm 3.641405579973e+00 true resid norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 61 KSP preconditioned resid norm 7.891395315495e-10 true resid norm 7.891395315494e-10 ||Ae||/||Ax|| 1.348744223425e-04 0 KSP preconditioned resid norm 2.675881805310e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.675881805310e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.453713947810e+00 true resid norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 1 KSP preconditioned resid norm 1.453713947810e+00 true resid norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 62 KSP preconditioned resid norm 7.510552985484e-10 true resid norm 7.510552985213e-10 ||Ae||/||Ax|| 1.283653213221e-04 0 KSP preconditioned resid norm 1.338840353148e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.338840353148e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.146220467265e+00 true resid norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 1 KSP preconditioned resid norm 1.146220467265e+00 true resid norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 63 KSP preconditioned resid norm 6.832740701908e-10 true resid norm 6.832740702025e-10 ||Ae||/||Ax|| 1.167806095574e-04 0 KSP preconditioned resid norm 2.478031412327e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.478031412327e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.547561676965e+00 true resid norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 1 KSP preconditioned resid norm 1.547561676965e+00 true resid norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 64 KSP preconditioned resid norm 6.312416928849e-10 true resid norm 6.312416928420e-10 ||Ae||/||Ax|| 1.078875854989e-04 0 KSP preconditioned resid norm 2.510304348429e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.510304348429e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.470709935588e+00 true resid norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 1 KSP preconditioned resid norm 1.470709935588e+00 true resid norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 65 KSP preconditioned resid norm 5.327354965414e-10 true resid norm 5.327354965097e-10 ||Ae||/||Ax|| 9.105156880435e-05 0 KSP preconditioned resid norm 2.476662569003e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.476662569003e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.310145576275e+00 true resid norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 1 KSP preconditioned resid norm 2.310145576275e+00 true resid norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 66 KSP preconditioned resid norm 4.469580732178e-10 true resid norm 4.469580732741e-10 ||Ae||/||Ax|| 7.639106841575e-05 0 KSP preconditioned resid norm 3.243520644840e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.243520644840e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.732155047053e+00 true resid norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 1 KSP preconditioned resid norm 2.732155047053e+00 true resid norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 67 KSP preconditioned resid norm 4.087445950732e-10 true resid norm 4.087445950372e-10 ||Ae||/||Ax|| 6.985987767337e-05 0 KSP preconditioned resid norm 4.570320195732e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.570320195732e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.087601985141e+00 true resid norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 1 KSP preconditioned resid norm 2.087601985141e+00 true resid norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 68 KSP preconditioned resid norm 3.841153386650e-10 true resid norm 3.841153386559e-10 ||Ae||/||Ax|| 6.565041078654e-05 0 KSP preconditioned resid norm 5.718860667569e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.718860667569e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.709934079219e+00 true resid norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 1 KSP preconditioned resid norm 1.709934079219e+00 true resid norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 69 KSP preconditioned resid norm 3.520264512665e-10 true resid norm 3.520264512287e-10 ||Ae||/||Ax|| 6.016599392193e-05 0 KSP preconditioned resid norm 4.537109521149e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.537109521149e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.937349360261e+00 true resid norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 1 KSP preconditioned resid norm 1.937349360261e+00 true resid norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 70 KSP preconditioned resid norm 3.149991658898e-10 true resid norm 3.149991658892e-10 ||Ae||/||Ax|| 5.383753929329e-05 0 KSP preconditioned resid norm 3.602956988994e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.602956988994e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.093057977304e+00 true resid norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 1 KSP preconditioned resid norm 2.093057977304e+00 true resid norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 71 KSP preconditioned resid norm 2.993160400510e-10 true resid norm 2.993160400417e-10 ||Ae||/||Ax|| 5.115708488107e-05 0 KSP preconditioned resid norm 3.336754379842e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.336754379842e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.284284263997e+00 true resid norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 1 KSP preconditioned resid norm 1.284284263997e+00 true resid norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 72 KSP preconditioned resid norm 2.895591295761e-10 true resid norm 2.895591295591e-10 ||Ae||/||Ax|| 4.948949934951e-05 0 KSP preconditioned resid norm 2.267067098558e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.267067098558e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.534973699953e-01 true resid norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 1 KSP preconditioned resid norm 8.534973699953e-01 true resid norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 73 KSP preconditioned resid norm 2.812425874590e-10 true resid norm 2.812425874308e-10 ||Ae||/||Ax|| 4.806809189165e-05 0 KSP preconditioned resid norm 1.144552955812e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.144552955812e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.664517534844e-01 true resid norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 1 KSP preconditioned resid norm 6.664517534844e-01 true resid norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 74 KSP preconditioned resid norm 2.735484153434e-10 true resid norm 2.735484153084e-10 ||Ae||/||Ax|| 4.675305573021e-05 0 KSP preconditioned resid norm 1.554702841568e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.554702841568e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.157380114110e-01 true resid norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 1 KSP preconditioned resid norm 7.157380114110e-01 true resid norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 75 KSP preconditioned resid norm 2.661574205602e-10 true resid norm 2.661574205227e-10 ||Ae||/||Ax|| 4.548983660049e-05 0 KSP preconditioned resid norm 4.199331451693e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.199331451693e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.261245258089e-01 true resid norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 1 KSP preconditioned resid norm 7.261245258089e-01 true resid norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 76 KSP preconditioned resid norm 2.534998275631e-10 true resid norm 2.534998275209e-10 ||Ae||/||Ax|| 4.332648591775e-05 0 KSP preconditioned resid norm 3.743395182510e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.743395182510e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.990417749799e-01 true resid norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 1 KSP preconditioned resid norm 8.990417749799e-01 true resid norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 77 KSP preconditioned resid norm 2.388049684536e-10 true resid norm 2.388049684532e-10 ||Ae||/||Ax|| 4.081493941813e-05 0 KSP preconditioned resid norm 1.894155910280e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.894155910280e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.684190700742e-01 true resid norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 1 KSP preconditioned resid norm 8.684190700742e-01 true resid norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 78 KSP preconditioned resid norm 2.286512776556e-10 true resid norm 2.286512776733e-10 ||Ae||/||Ax|| 3.907953886624e-05 0 KSP preconditioned resid norm 1.276207551837e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.276207551837e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.355779104233e-01 true resid norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 1 KSP preconditioned resid norm 5.355779104233e-01 true resid norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 79 KSP preconditioned resid norm 2.074225711250e-10 true resid norm 2.074225711493e-10 ||Ae||/||Ax|| 3.545127109477e-05 0 KSP preconditioned resid norm 1.200062868567e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.200062868567e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.893976477109e-01 true resid norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 1 KSP preconditioned resid norm 6.893976477109e-01 true resid norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 80 KSP preconditioned resid norm 1.878908772984e-10 true resid norm 1.878908773103e-10 ||Ae||/||Ax|| 3.211304532026e-05 0 KSP preconditioned resid norm 1.387129119694e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.387129119694e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.742548405622e-01 true resid norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 1 KSP preconditioned resid norm 8.742548405622e-01 true resid norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 81 KSP preconditioned resid norm 1.720957363240e-10 true resid norm 1.720957363027e-10 ||Ae||/||Ax|| 2.941344603009e-05 0 KSP preconditioned resid norm 1.602115661552e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.602115661552e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.392172521059e-01 true resid norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 1 KSP preconditioned resid norm 8.392172521059e-01 true resid norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 82 KSP preconditioned resid norm 1.584755992063e-10 true resid norm 1.584755991999e-10 ||Ae||/||Ax|| 2.708558378201e-05 0 KSP preconditioned resid norm 1.601197866084e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.601197866084e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.359479912386e-01 true resid norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 1 KSP preconditioned resid norm 8.359479912386e-01 true resid norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 83 KSP preconditioned resid norm 1.418702609113e-10 true resid norm 1.418702608829e-10 ||Ae||/||Ax|| 2.424751101569e-05 0 KSP preconditioned resid norm 1.499465989768e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.499465989768e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.045754716297e+00 true resid norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 1 KSP preconditioned resid norm 1.045754716297e+00 true resid norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 84 KSP preconditioned resid norm 1.271329562402e-10 true resid norm 1.271329562130e-10 ||Ae||/||Ax|| 2.172870999918e-05 0 KSP preconditioned resid norm 1.634503776014e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.634503776014e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.222338373864e+00 true resid norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 1 KSP preconditioned resid norm 1.222338373864e+00 true resid norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 85 KSP preconditioned resid norm 1.170830928689e-10 true resid norm 1.170830928467e-10 ||Ae||/||Ax|| 2.001105493065e-05 0 KSP preconditioned resid norm 2.189478982761e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.189478982761e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.257863289529e+00 true resid norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 1 KSP preconditioned resid norm 1.257863289529e+00 true resid norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 86 KSP preconditioned resid norm 1.072794488701e-10 true resid norm 1.072794488559e-10 ||Ae||/||Ax|| 1.833548202211e-05 0 KSP preconditioned resid norm 3.043239239438e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.043239239438e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.508125411857e+00 true resid norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 1 KSP preconditioned resid norm 1.508125411857e+00 true resid norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 87 KSP preconditioned resid norm 9.310419868753e-11 true resid norm 9.310419867003e-11 ||Ae||/||Ax|| 1.591274357860e-05 0 KSP preconditioned resid norm 2.707043890291e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.707043890291e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.156895340838e+00 true resid norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 1 KSP preconditioned resid norm 2.156895340838e+00 true resid norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 88 KSP preconditioned resid norm 7.852841829285e-11 true resid norm 7.852841828475e-11 ||Ae||/||Ax|| 1.342154920668e-05 0 KSP preconditioned resid norm 2.582289916941e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.582289916941e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.382288228383e+00 true resid norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 1 KSP preconditioned resid norm 2.382288228383e+00 true resid norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 89 KSP preconditioned resid norm 6.494099865988e-11 true resid norm 6.494099869486e-11 ||Ae||/||Ax|| 1.109927881590e-05 0 KSP preconditioned resid norm 2.588940215193e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.588940215193e+00 true resid norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.412871891541e+00 true resid norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 1 KSP preconditioned resid norm 2.412871891541e+00 true resid norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 90 KSP preconditioned resid norm 5.586819688621e-11 true resid norm 5.586819689526e-11 ||Ae||/||Ax|| 9.548616540312e-06 2 SNES Function norm 5.587280798316e-11 SNES Object: type: ls line search variant: SNESLineSearchCubic alpha=0.0001, maxstep=1e+08, minlambda=1e-12 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=107 total number of function evaluations=3 KSP Object: type: fgmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 right preconditioning PC Object: type: mg MG: type is FULL, levels=1 cycles=v Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- KSP Object:(mg_levels_0_) type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(mg_levels_0_) type: ilu ILU: 0 levels of fill ILU: factor fill ratio allocated 1 ILU: tolerance for zero pivot 1e-12 ILU: using diagonal shift to prevent zero pivot ILU: using diagonal shift on blocks to prevent zero pivot out-of-place factorization matrix ordering: natural ILU: factor fill ratio needed 1 Factored matrix follows Matrix Object: type=seqaij, rows=39366, cols=39366 package used to perform factorization: petsc total: nonzeros=2125764, allocated nonzeros=2125764 using I-node routines: found 19683 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=39366, cols=39366 total: nonzeros=2125764, allocated nonzeros=2125764 using I-node routines: found 19683 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=39366, cols=39366 total: nonzeros=2125764, allocated nonzeros=2125764 using I-node routines: found 19683 nodes, limit used is 5 Thanks again, Hal Barry Smith wrote: > > Please run with > > -ksp_monitor_true_residual -snes_monitor > -mg_levels_ksp_monitor_true_residua -mg_coarse_ksp_type gmres > -mg_coarse_ksp_monitor_true_residual -snes_view > > and send the output for one time-step. > > Barry > > > On Aug 26, 2009, at 2:34 PM, Hal Finkel wrote: > >> Hello, >> >> I've constructed a evolution scheme for a time-dependent 3D PDE system >> using the DMMG layer. The core of the code is essentially a loop >> around DMMGSolve in the spirit of snes/examples/tutorials/ex29.c. If I >> only run with one level, then the code runs and produces sensible >> results. Passing the option -mg_levels_0_ksp_type bcgsl increases the >> solver speed by about 30%, and -mg_levels_0_pc_type fieldsplit gives >> me another 5% improvement. Does that say something about the structure >> of my problem? >> >> If I run with two levels, the coarse solve seems to work, but the fine >> grid's KSP never seems to complete. If I run with >> -mg_levels_1_ksp_monitor_true_residual, I can see the residual does >> not seem to be converging at all: >> >> 0 KSP preconditioned resid norm 1.979915874387e+02 true resid >> norm 1.889855551272e+04 ||Ae||/||Ax|| 1.889855551272e+04 >> 0 KSP preconditioned resid norm 3.833279709488e+06 true resid >> norm 3.665290817713e+08 ||Ae||/||Ax|| 3.665290817713e+08 >> 0 KSP preconditioned resid norm 5.080465654647e+01 true resid >> norm 8.284057252719e+03 ||Ae||/||Ax|| 8.284057252719e+03 >> 1 KSP preconditioned resid norm 6.057649668915e+00 true resid >> norm 5.136009463814e+02 ||Ae||/||Ax|| 5.136009463814e+02 >> 0 KSP preconditioned resid norm 5.511785811223e+04 true resid >> norm 1.055162492334e+07 ||Ae||/||Ax|| 1.055162492334e+07 >> 0 KSP preconditioned resid norm 2.164672340919e+01 true resid >> norm 2.999406329054e+04 ||Ae||/||Ax|| 2.999406329054e+04 >> 0 KSP preconditioned resid norm 5.275983407134e+05 true resid >> norm 9.196351032040e+08 ||Ae||/||Ax|| 9.196351032040e+08 >> 0 KSP preconditioned resid norm 3.338344860439e+01 true resid >> norm 5.133553722923e+03 ||Ae||/||Ax|| 5.133553722923e+03 >> 1 KSP preconditioned resid norm 4.306407436023e+00 true resid >> norm 3.361745947223e+02 ||Ae||/||Ax|| 3.361745947223e+02 >> ... >> >> If I run with -pc_mg_monitor I see, after the course solve, only >> things like: >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 1.979915874387e+02 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 3.833279709488e+06 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 5.080465654647e+01 >> 1 KSP Residual norm 6.057649668915e+00 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 5.511785811223e+04 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 2.164672340919e+01 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 5.275983407134e+05 >> Residual norms for mg_levels_1_ solve. >> 0 KSP Residual norm 3.338344860439e+01 >> 1 KSP Residual norm 4.306407436023e+00 >> ... >> >> running with -info shows: >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >> computing 2-norm of preconditioned RHS >> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >> hand size norm 0.0066162, current residual norm 23282.7 at iteration 0 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >> computing 2-norm of preconditioned RHS >> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >> hand size norm 0.00287711, current residual norm 38.2205 at iteration 0 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >> computing 2-norm of preconditioned RHS >> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >> hand size norm 0.00287711, current residual norm 809191 at iteration 0 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >> computing 2-norm of preconditioned RHS >> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >> hand size norm 0.00970186, current residual norm 186.983 at iteration 0 >> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >> computing 2-norm of preconditioned RHS >> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >> hand size norm 0.00970186, current residual norm 3.62292e+06 at >> iteration 0 >> >> What is a good way to debug this? >> >> No matter how I run, I've arranged the code so that the fine grid is >> *always* the same size; it has to be because the initial conditions >> are loaded from a file. >> >> In general, however, I don't understand exactly what the mg >> preconditioner is doing. Is there a writeup somewhere which is more >> verbose than the manual? In the one-level case, how are the results of >> the solve used to precondition the Jacobian? >> >> In an 1-processor, 1-level case, running with -snes_view gives: >> SNES Object: >> type: ls >> line search variant: SNESLineSearchCubic >> alpha=0.0001, maxstep=1e+08, minlambda=1e-12 >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=36 >> total number of function evaluations=3 >> KSP Object: >> type: fgmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> right preconditioning >> PC Object: >> type: mg >> MG: type is FULL, levels=1 cycles=v >> Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- >> KSP Object:(mg_levels_0_) >> type: bcgsl >> BCGSL: Ell = 2 >> BCGSL: Delta = 0 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_0_) >> type: fieldsplit >> FieldSplit with MULTIPLICATIVE composition: total splits = 2, >> blocksize = 2 >> Solver info for each split is in the following KSP objects: >> Split number 0 Fields 0 >> KSP Object:(mg_levels_0_fieldsplit_0_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_0_fieldsplit_0_) >> type: ilu >> ILU: 0 levels of fill >> ILU: factor fill ratio allocated 1 >> ILU: tolerance for zero pivot 1e-12 >> ILU: using diagonal shift to prevent zero pivot >> ILU: using diagonal shift on blocks to prevent zero pivot >> out-of-place factorization >> matrix ordering: natural >> ILU: factor fill ratio needed 1 >> Factored matrix follows >> Matrix Object: >> type=seqaij, rows=19683, cols=19683 >> package used to perform factorization: petsc >> total: nonzeros=531441, allocated nonzeros=531441 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=19683, cols=19683 >> total: nonzeros=531441, allocated nonzeros=531441 >> not using I-node routines >> Split number 1 Fields 1 >> KSP Object:(mg_levels_0_fieldsplit_1_) >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_0_fieldsplit_1_) >> type: ilu >> ILU: 0 levels of fill >> ILU: factor fill ratio allocated 1 >> ILU: tolerance for zero pivot 1e-12 >> ILU: using diagonal shift to prevent zero pivot >> ILU: using diagonal shift on blocks to prevent zero pivot >> out-of-place factorization >> matrix ordering: natural >> ILU: factor fill ratio needed 1 >> Factored matrix follows >> Matrix Object: >> type=seqaij, rows=19683, cols=19683 >> package used to perform factorization: petsc >> total: nonzeros=531441, allocated nonzeros=531441 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=19683, cols=19683 >> total: nonzeros=531441, allocated nonzeros=531441 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=39366, cols=39366 >> total: nonzeros=2125764, allocated nonzeros=2125764 >> using I-node routines: found 19683 nodes, limit used is 5 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=39366, cols=39366 >> total: nonzeros=2125764, allocated nonzeros=2125764 >> using I-node routines: found 19683 nodes, limit used is 5 >> >> Thank you in advance, >> Hal From bsmith at mcs.anl.gov Wed Aug 26 15:49:39 2009 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 26 Aug 2009 15:49:39 -0500 Subject: Multigrid Questions In-Reply-To: <4A959F07.9010506@yale.edu> References: <4A958E3C.2070400@yale.edu> <2EF3551E-FEA7-4A09-BE94-41BA88E0190B@mcs.anl.gov> <4A959F07.9010506@yale.edu> Message-ID: <1570E834-D71E-4F0B-B4C4-90C926A2988D@mcs.anl.gov> No, I want it on the nonworking two-level case. Isn't that your goal to do multigrid? If one level is working and you are happy with it then why send the email? Barry On Aug 26, 2009, at 3:45 PM, Hal Finkel wrote: > Barry, > > Here is the output with those options in the (working) 1-level case > on 1 cpu. Is that what you wanted? > > 0 SNES Function norm 6.643976542462e-01 > 0 KSP preconditioned resid norm 6.643976542462e-01 true resid > norm 6.643976542462e-01 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.438911631652e-03 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.438911631652e-03 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.141505633982e-04 true resid > norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 > 1 KSP preconditioned resid norm 5.141505633982e-04 true resid > norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 > 1 KSP preconditioned resid norm 4.790430472989e-02 true resid > norm 4.790430472989e-02 ||Ae||/||Ax|| 7.210185710881e-02 > 0 KSP preconditioned resid norm 8.314136308720e-03 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.314136308720e-03 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.510029589596e-03 true resid > norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 > 1 KSP preconditioned resid norm 4.510029589596e-03 true resid > norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 > 2 KSP preconditioned resid norm 8.487458086567e-03 true resid > norm 8.487458086567e-03 ||Ae||/||Ax|| 1.277466594339e-02 > 0 KSP preconditioned resid norm 6.119493660032e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 6.119493660032e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.236002849750e-02 true resid > norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 > 1 KSP preconditioned resid norm 2.236002849750e-02 true resid > norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 > 3 KSP preconditioned resid norm 5.965466932886e-03 true resid > norm 5.965466932886e-03 ||Ae||/||Ax|| 8.978759775506e-03 > 0 KSP preconditioned resid norm 1.765708232662e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.765708232662e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.422901024605e-02 true resid > norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 > 1 KSP preconditioned resid norm 2.422901024605e-02 true resid > norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 > 4 KSP preconditioned resid norm 4.622977909431e-03 true resid > norm 4.622977909431e-03 ||Ae||/||Ax|| 6.958149054087e-03 > 0 KSP preconditioned resid norm 8.428635776507e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.428635776507e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.071459298111e-02 true resid > norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 > 1 KSP preconditioned resid norm 1.071459298111e-02 true resid > norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 > 5 KSP preconditioned resid norm 2.949006884559e-03 true resid > norm 2.949006884560e-03 ||Ae||/||Ax|| 4.438617243322e-03 > 0 KSP preconditioned resid norm 4.221520918573e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.221520918573e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.795668593850e-02 true resid > norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 > 1 KSP preconditioned resid norm 1.795668593850e-02 true resid > norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 > 6 KSP preconditioned resid norm 1.327402359192e-03 true resid > norm 1.327402359192e-03 ||Ae||/||Ax|| 1.997903440370e-03 > 0 KSP preconditioned resid norm 3.239330680409e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.239330680409e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.103583195381e-02 true resid > norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 > 1 KSP preconditioned resid norm 1.103583195381e-02 true resid > norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 > 7 KSP preconditioned resid norm 5.579319891254e-04 true resid > norm 5.579319891255e-04 ||Ae||/||Ax|| 8.397561092512e-04 > 0 KSP preconditioned resid norm 2.166256693843e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.166256693843e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.527142494434e-02 true resid > norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 > 1 KSP preconditioned resid norm 2.527142494434e-02 true resid > norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 > 8 KSP preconditioned resid norm 1.841437265702e-04 true resid > norm 1.841437265703e-04 ||Ae||/||Ax|| 2.771589053535e-04 > 0 KSP preconditioned resid norm 1.097958713832e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.097958713832e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.070826208815e-02 true resid > norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 > 1 KSP preconditioned resid norm 4.070826208815e-02 true resid > norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 > 9 KSP preconditioned resid norm 6.545738966234e-05 true resid > norm 6.545738966232e-05 ||Ae||/||Ax|| 9.852140392726e-05 > 0 KSP preconditioned resid norm 9.668719808206e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.668719808206e-02 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.944112488019e-02 true resid > norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 > 1 KSP preconditioned resid norm 7.944112488019e-02 true resid > norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 > 10 KSP preconditioned resid norm 4.125461342831e-05 true resid > norm 4.125461342821e-05 ||Ae||/||Ax|| 6.209325569492e-05 > 0 KSP preconditioned resid norm 3.008151203349e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.008151203349e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.247738676584e-02 true resid > norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 > 1 KSP preconditioned resid norm 7.247738676584e-02 true resid > norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 > 11 KSP preconditioned resid norm 3.280778128691e-05 true resid > norm 3.280778128689e-05 ||Ae||/||Ax|| 4.937973678446e-05 > 0 KSP preconditioned resid norm 9.091470733399e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.091470733399e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.215403990109e-01 true resid > norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 > 1 KSP preconditioned resid norm 1.215403990109e-01 true resid > norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 > 12 KSP preconditioned resid norm 2.968550831459e-05 true resid > norm 2.968550831435e-05 ||Ae||/||Ax|| 4.468033281669e-05 > 0 KSP preconditioned resid norm 1.024246289230e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.024246289230e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.302673603327e-02 true resid > norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 > 1 KSP preconditioned resid norm 9.302673603327e-02 true resid > norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 > 13 KSP preconditioned resid norm 2.521514028222e-05 true resid > norm 2.521514028196e-05 ||Ae||/||Ax|| 3.795188035479e-05 > 0 KSP preconditioned resid norm 5.990912160630e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.990912160630e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.781990197263e-02 true resid > norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 > 1 KSP preconditioned resid norm 9.781990197263e-02 true resid > norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 > 14 KSP preconditioned resid norm 1.978375072845e-05 true resid > norm 1.978375072804e-05 ||Ae||/||Ax|| 2.977697257298e-05 > 0 KSP preconditioned resid norm 1.008331187053e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.008331187053e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.294954965258e-01 true resid > norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 > 1 KSP preconditioned resid norm 2.294954965258e-01 true resid > norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 > 15 KSP preconditioned resid norm 1.488645974384e-05 true resid > norm 1.488645974344e-05 ||Ae||/||Ax|| 2.240594867893e-05 > 0 KSP preconditioned resid norm 1.849227047229e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.849227047229e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.919984907088e-01 true resid > norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 > 1 KSP preconditioned resid norm 2.919984907088e-01 true resid > norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 > 16 KSP preconditioned resid norm 7.094849419546e-06 true resid > norm 7.094849419588e-06 ||Ae||/||Ax|| 1.067861900813e-05 > 0 KSP preconditioned resid norm 1.388766130430e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.388766130430e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.878119571370e-01 true resid > norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 > 1 KSP preconditioned resid norm 3.878119571370e-01 true resid > norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 > 17 KSP preconditioned resid norm 4.097305136804e-06 true resid > norm 4.097305136552e-06 ||Ae||/||Ax|| 6.166947023919e-06 > 1 SNES Function norm 5.850920566283e-06 > 0 KSP preconditioned resid norm 5.850920566283e-06 true resid > norm 5.850920566283e-06 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.276021960717e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.276021960717e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.809774037292e-01 true resid > norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 > 1 KSP preconditioned resid norm 3.809774037292e-01 true resid > norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 > 1 KSP preconditioned resid norm 3.947856942305e-06 true resid > norm 3.947856942305e-06 ||Ae||/||Ax|| 6.747411621096e-01 > 0 KSP preconditioned resid norm 2.497905632896e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.497905632896e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.608088417629e-01 true resid > norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 > 1 KSP preconditioned resid norm 5.608088417629e-01 true resid > norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 > 2 KSP preconditioned resid norm 3.305771768709e-06 true resid > norm 3.305771768709e-06 ||Ae||/||Ax|| 5.650002817948e-01 > 0 KSP preconditioned resid norm 7.301425383804e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.301425383804e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.476223530528e-01 true resid > norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 > 1 KSP preconditioned resid norm 3.476223530528e-01 true resid > norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 > 3 KSP preconditioned resid norm 3.007329187251e-06 true resid > norm 3.007329187251e-06 ||Ae||/||Ax|| 5.139924825815e-01 > 0 KSP preconditioned resid norm 2.880397337024e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.880397337024e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.471977831965e-01 true resid > norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 > 1 KSP preconditioned resid norm 4.471977831965e-01 true resid > norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 > 4 KSP preconditioned resid norm 3.001204159469e-06 true resid > norm 3.001204159469e-06 ||Ae||/||Ax|| 5.129456340194e-01 > 0 KSP preconditioned resid norm 4.619989941290e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.619989941290e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.243648576810e-01 true resid > norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 > 1 KSP preconditioned resid norm 3.243648576810e-01 true resid > norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 > 5 KSP preconditioned resid norm 3.001115709599e-06 true resid > norm 3.001115709599e-06 ||Ae||/||Ax|| 5.129305167624e-01 > 0 KSP preconditioned resid norm 4.829994865746e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.829994865746e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.325734548745e-01 true resid > norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 > 1 KSP preconditioned resid norm 4.325734548745e-01 true resid > norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 > 6 KSP preconditioned resid norm 2.974277803753e-06 true resid > norm 2.974277803753e-06 ||Ae||/||Ax|| 5.083435623606e-01 > 0 KSP preconditioned resid norm 1.381129102428e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.381129102428e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.098881160663e-01 true resid > norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 > 1 KSP preconditioned resid norm 3.098881160663e-01 true resid > norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 > 7 KSP preconditioned resid norm 2.816838388951e-06 true resid > norm 2.816838388951e-06 ||Ae||/||Ax|| 4.814350762483e-01 > 0 KSP preconditioned resid norm 1.559573417426e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.559573417426e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.532723445272e-01 true resid > norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 > 1 KSP preconditioned resid norm 2.532723445272e-01 true resid > norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 > 8 KSP preconditioned resid norm 2.572324455447e-06 true resid > norm 2.572324455447e-06 ||Ae||/||Ax|| 4.396443989123e-01 > 0 KSP preconditioned resid norm 5.946831260059e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.946831260059e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.929968207615e-01 true resid > norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 > 1 KSP preconditioned resid norm 3.929968207615e-01 true resid > norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 > 9 KSP preconditioned resid norm 2.177455315353e-06 true resid > norm 2.177455315353e-06 ||Ae||/||Ax|| 3.721560206954e-01 > 0 KSP preconditioned resid norm 5.315723299063e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.315723299063e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.628504511585e-01 true resid > norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 > 1 KSP preconditioned resid norm 3.628504511585e-01 true resid > norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 > 10 KSP preconditioned resid norm 1.409067985559e-06 true resid > norm 1.409067985559e-06 ||Ae||/||Ax|| 2.408284251334e-01 > 0 KSP preconditioned resid norm 1.522258580690e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.522258580690e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.414238177111e-01 true resid > norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 > 1 KSP preconditioned resid norm 2.414238177111e-01 true resid > norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 > 11 KSP preconditioned resid norm 1.001503517728e-06 true resid > norm 1.001503517728e-06 ||Ae||/||Ax|| 1.711702468667e-01 > 0 KSP preconditioned resid norm 8.244553153983e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.244553153983e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.244218748685e-01 true resid > norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 > 1 KSP preconditioned resid norm 2.244218748685e-01 true resid > norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 > 12 KSP preconditioned resid norm 7.556215980081e-07 true resid > norm 7.556215980082e-07 ||Ae||/||Ax|| 1.291457625254e-01 > 0 KSP preconditioned resid norm 1.857974243745e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.857974243745e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.451523414794e-01 true resid > norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 > 1 KSP preconditioned resid norm 3.451523414794e-01 true resid > norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 > 13 KSP preconditioned resid norm 5.872140840954e-07 true resid > norm 5.872140840955e-07 ||Ae||/||Ax|| 1.003626826656e-01 > 0 KSP preconditioned resid norm 2.611648570586e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.611648570586e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.740043981340e-01 true resid > norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 > 1 KSP preconditioned resid norm 2.740043981340e-01 true resid > norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 > 14 KSP preconditioned resid norm 4.352350316840e-07 true resid > norm 4.352350316841e-07 ||Ae||/||Ax|| 7.438744497613e-02 > 0 KSP preconditioned resid norm 2.003016340281e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.003016340281e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.073409352245e-01 true resid > norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 > 1 KSP preconditioned resid norm 4.073409352245e-01 true resid > norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 > 15 KSP preconditioned resid norm 3.052867274622e-07 true resid > norm 3.052867274624e-07 ||Ae||/||Ax|| 5.217755462648e-02 > 0 KSP preconditioned resid norm 1.286639096168e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.286639096168e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.656621503377e-01 true resid > norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 > 1 KSP preconditioned resid norm 3.656621503377e-01 true resid > norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 > 16 KSP preconditioned resid norm 1.955997188783e-07 true resid > norm 1.955997188782e-07 ||Ae||/||Ax|| 3.343058868468e-02 > 0 KSP preconditioned resid norm 1.335903581256e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.335903581256e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.001895431683e-01 true resid > norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 > 1 KSP preconditioned resid norm 3.001895431683e-01 true resid > norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 > 17 KSP preconditioned resid norm 1.567782688997e-07 true resid > norm 1.567782688996e-07 ||Ae||/||Ax|| 2.679548750039e-02 > 0 KSP preconditioned resid norm 1.651696143482e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.651696143482e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.713972348795e-01 true resid > norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 > 1 KSP preconditioned resid norm 2.713972348795e-01 true resid > norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 > 18 KSP preconditioned resid norm 1.010718356377e-07 true resid > norm 1.010718356378e-07 ||Ae||/||Ax|| 1.727451851256e-02 > 0 KSP preconditioned resid norm 1.011081720758e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.011081720758e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.527570671391e-01 true resid > norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 > 1 KSP preconditioned resid norm 5.527570671391e-01 true resid > norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 > 19 KSP preconditioned resid norm 8.997938714701e-08 true resid > norm 8.997938714709e-08 ||Ae||/||Ax|| 1.537867180519e-02 > 0 KSP preconditioned resid norm 1.537108246495e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.537108246495e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.298772065070e-01 true resid > norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 > 1 KSP preconditioned resid norm 4.298772065070e-01 true resid > norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 > 20 KSP preconditioned resid norm 8.711914509357e-08 true resid > norm 8.711914509361e-08 ||Ae||/||Ax|| 1.488981846646e-02 > 0 KSP preconditioned resid norm 1.923097362850e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.923097362850e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.075948671586e-01 true resid > norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 > 1 KSP preconditioned resid norm 3.075948671586e-01 true resid > norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 > 21 KSP preconditioned resid norm 8.323067409578e-08 true resid > norm 8.323067409577e-08 ||Ae||/||Ax|| 1.422522715065e-02 > 0 KSP preconditioned resid norm 9.819276869885e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.819276869885e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.279383354509e-01 true resid > norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 > 1 KSP preconditioned resid norm 4.279383354509e-01 true resid > norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 > 22 KSP preconditioned resid norm 7.909034553981e-08 true resid > norm 7.909034553913e-08 ||Ae||/||Ax|| 1.351759003445e-02 > 0 KSP preconditioned resid norm 9.582482498733e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.582482498733e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.872542232901e-01 true resid > norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 > 1 KSP preconditioned resid norm 4.872542232901e-01 true resid > norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 > 23 KSP preconditioned resid norm 7.583943619887e-08 true resid > norm 7.583943619831e-08 ||Ae||/||Ax|| 1.296196646992e-02 > 0 KSP preconditioned resid norm 1.129055084162e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.129055084162e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.940290407027e-01 true resid > norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 > 1 KSP preconditioned resid norm 3.940290407027e-01 true resid > norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 > 24 KSP preconditioned resid norm 7.198974227007e-08 true resid > norm 7.198974226927e-08 ||Ae||/||Ax|| 1.230400266996e-02 > 0 KSP preconditioned resid norm 7.922461164088e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.922461164088e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.568631157402e-01 true resid > norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 > 1 KSP preconditioned resid norm 3.568631157402e-01 true resid > norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 > 25 KSP preconditioned resid norm 6.969681693595e-08 true resid > norm 6.969681693509e-08 ||Ae||/||Ax|| 1.191211129010e-02 > 0 KSP preconditioned resid norm 8.506106408028e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.506106408028e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.674832821714e-01 true resid > norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 > 1 KSP preconditioned resid norm 2.674832821714e-01 true resid > norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 > 26 KSP preconditioned resid norm 6.713093474885e-08 true resid > norm 6.713093474791e-08 ||Ae||/||Ax|| 1.147356796036e-02 > 0 KSP preconditioned resid norm 7.107668950850e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.107668950850e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.080542740675e-01 true resid > norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 > 1 KSP preconditioned resid norm 3.080542740675e-01 true resid > norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 > 27 KSP preconditioned resid norm 6.316951623357e-08 true resid > norm 6.316951623251e-08 ||Ae||/||Ax|| 1.079650894537e-02 > 0 KSP preconditioned resid norm 9.816834307897e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.816834307897e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.035636714372e-01 true resid > norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 > 1 KSP preconditioned resid norm 3.035636714372e-01 true resid > norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 > 28 KSP preconditioned resid norm 5.685161562141e-08 true resid > norm 5.685161562024e-08 ||Ae||/||Ax|| 9.716695856008e-03 > 0 KSP preconditioned resid norm 1.017862349675e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.017862349675e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.438638918414e-01 true resid > norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 > 1 KSP preconditioned resid norm 3.438638918414e-01 true resid > norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 > 29 KSP preconditioned resid norm 5.170566864610e-08 true resid > norm 5.170566864501e-08 ||Ae||/||Ax|| 8.837185201757e-03 > 0 KSP preconditioned resid norm 8.897697988733e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.897697988733e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.213181899588e-01 true resid > norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 > 1 KSP preconditioned resid norm 3.213181899588e-01 true resid > norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 > 30 KSP preconditioned resid norm 4.885061460285e-08 true resid > norm 4.885061460285e-08 ||Ae||/||Ax|| 8.349218563034e-03 > 0 KSP preconditioned resid norm 7.100457326024e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.100457326024e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.955812925302e-01 true resid > norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 > 1 KSP preconditioned resid norm 6.955812925302e-01 true resid > norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 > 31 KSP preconditioned resid norm 4.725291213013e-08 true resid > norm 4.725291213007e-08 ||Ae||/||Ax|| 8.076149999775e-03 > 0 KSP preconditioned resid norm 4.185203199010e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.185203199010e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.196403938530e-01 true resid > norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 > 1 KSP preconditioned resid norm 2.196403938530e-01 true resid > norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 > 32 KSP preconditioned resid norm 4.547823851359e-08 true resid > norm 4.547823851362e-08 ||Ae||/||Ax|| 7.772834718641e-03 > 0 KSP preconditioned resid norm 6.581109982585e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 6.581109982585e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.128181724937e-01 true resid > norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 > 1 KSP preconditioned resid norm 3.128181724937e-01 true resid > norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 > 33 KSP preconditioned resid norm 4.468618222477e-08 true resid > norm 4.468618222483e-08 ||Ae||/||Ax|| 7.637461783764e-03 > 0 KSP preconditioned resid norm 8.120341282592e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.120341282592e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.218250119940e-01 true resid > norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 > 1 KSP preconditioned resid norm 2.218250119940e-01 true resid > norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 > 34 KSP preconditioned resid norm 4.314354409778e-08 true resid > norm 4.314354409782e-08 ||Ae||/||Ax|| 7.373804448217e-03 > 0 KSP preconditioned resid norm 5.778947005677e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.778947005677e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.877938034235e-01 true resid > norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 > 1 KSP preconditioned resid norm 3.877938034235e-01 true resid > norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 > 35 KSP preconditioned resid norm 4.111383309670e-08 true resid > norm 4.111383309672e-08 ||Ae||/||Ax|| 7.026899892239e-03 > 0 KSP preconditioned resid norm 7.589832929724e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.589832929724e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.735429451288e-01 true resid > norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 > 1 KSP preconditioned resid norm 2.735429451288e-01 true resid > norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 > 36 KSP preconditioned resid norm 3.896413302095e-08 true resid > norm 3.896413302095e-08 ||Ae||/||Ax|| 6.659487610460e-03 > 0 KSP preconditioned resid norm 9.173185113175e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.173185113175e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.052187383165e-01 true resid > norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 > 1 KSP preconditioned resid norm 3.052187383165e-01 true resid > norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 > 37 KSP preconditioned resid norm 3.770537943429e-08 true resid > norm 3.770537943431e-08 ||Ae||/||Ax|| 6.444349911635e-03 > 0 KSP preconditioned resid norm 1.137492649970e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.137492649970e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.044329763751e-01 true resid > norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 > 1 KSP preconditioned resid norm 3.044329763751e-01 true resid > norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 > 38 KSP preconditioned resid norm 3.694716763304e-08 true resid > norm 3.694716763306e-08 ||Ae||/||Ax|| 6.314761448988e-03 > 0 KSP preconditioned resid norm 1.223189932062e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.223189932062e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.107514686273e-01 true resid > norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 > 1 KSP preconditioned resid norm 2.107514686273e-01 true resid > norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 > 39 KSP preconditioned resid norm 3.634742486594e-08 true resid > norm 3.634742486593e-08 ||Ae||/||Ax|| 6.212257448064e-03 > 0 KSP preconditioned resid norm 1.192363148419e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.192363148419e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.860271918718e-01 true resid > norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 > 1 KSP preconditioned resid norm 2.860271918718e-01 true resid > norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 > 40 KSP preconditioned resid norm 3.561782722372e-08 true resid > norm 3.561782722374e-08 ||Ae||/||Ax|| 6.087559525075e-03 > 0 KSP preconditioned resid norm 8.881324685300e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 8.881324685300e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.642271412468e-01 true resid > norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 > 1 KSP preconditioned resid norm 2.642271412468e-01 true resid > norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 > 41 KSP preconditioned resid norm 3.506823410956e-08 true resid > norm 3.506823410956e-08 ||Ae||/||Ax|| 5.993626765614e-03 > 0 KSP preconditioned resid norm 9.513827527161e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 9.513827527161e-01 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.263980500382e-01 true resid > norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 > 1 KSP preconditioned resid norm 2.263980500382e-01 true resid > norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 > 42 KSP preconditioned resid norm 3.466236527477e-08 true resid > norm 3.466236527477e-08 ||Ae||/||Ax|| 5.924258393546e-03 > 0 KSP preconditioned resid norm 1.407625976444e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.407625976444e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.841824427477e-01 true resid > norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 > 1 KSP preconditioned resid norm 1.841824427477e-01 true resid > norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 > 43 KSP preconditioned resid norm 3.394294635279e-08 true resid > norm 3.394294635284e-08 ||Ae||/||Ax|| 5.801300148978e-03 > 0 KSP preconditioned resid norm 1.536263909707e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.536263909707e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.972519671057e-01 true resid > norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 > 1 KSP preconditioned resid norm 2.972519671057e-01 true resid > norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 > 44 KSP preconditioned resid norm 3.248296724023e-08 true resid > norm 3.248296724025e-08 ||Ae||/||Ax|| 5.551770336352e-03 > 0 KSP preconditioned resid norm 1.783804105384e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.783804105384e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.722881763352e-01 true resid > norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 > 1 KSP preconditioned resid norm 3.722881763352e-01 true resid > norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 > 45 KSP preconditioned resid norm 2.950320556254e-08 true resid > norm 2.950320556258e-08 ||Ae||/||Ax|| 5.042489507138e-03 > 0 KSP preconditioned resid norm 2.201496546447e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.201496546447e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.087649817784e-01 true resid > norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 > 1 KSP preconditioned resid norm 4.087649817784e-01 true resid > norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 > 46 KSP preconditioned resid norm 2.319584580786e-08 true resid > norm 2.319584580791e-08 ||Ae||/||Ax|| 3.964477990280e-03 > 0 KSP preconditioned resid norm 1.736355434315e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.736355434315e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.867386399797e-01 true resid > norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 > 1 KSP preconditioned resid norm 6.867386399797e-01 true resid > norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 > 47 KSP preconditioned resid norm 2.008430277340e-08 true resid > norm 2.008430277343e-08 ||Ae||/||Ax|| 3.432673977693e-03 > 0 KSP preconditioned resid norm 1.488127968937e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.488127968937e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.516786708506e-01 true resid > norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 > 1 KSP preconditioned resid norm 5.516786708506e-01 true resid > norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 > 48 KSP preconditioned resid norm 1.854206408497e-08 true resid > norm 1.854206408497e-08 ||Ae||/||Ax|| 3.169084911497e-03 > 0 KSP preconditioned resid norm 1.380328436388e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.380328436388e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.575661767087e-01 true resid > norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 > 1 KSP preconditioned resid norm 4.575661767087e-01 true resid > norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 > 49 KSP preconditioned resid norm 1.665547510085e-08 true resid > norm 1.665547510085e-08 ||Ae||/||Ax|| 2.846641808270e-03 > 0 KSP preconditioned resid norm 2.313536103290e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.313536103290e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.305138630648e-01 true resid > norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 > 1 KSP preconditioned resid norm 5.305138630648e-01 true resid > norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 > 50 KSP preconditioned resid norm 1.462283172240e-08 true resid > norm 1.462283172247e-08 ||Ae||/||Ax|| 2.499236070088e-03 > 0 KSP preconditioned resid norm 3.213123410447e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.213123410447e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.224884980518e-01 true resid > norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 > 1 KSP preconditioned resid norm 5.224884980518e-01 true resid > norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 > 51 KSP preconditioned resid norm 1.223657774649e-08 true resid > norm 1.223657774655e-08 ||Ae||/||Ax|| 2.091393586347e-03 > 0 KSP preconditioned resid norm 2.023552916685e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.023552916685e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.574715128775e-01 true resid > norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 > 1 KSP preconditioned resid norm 5.574715128775e-01 true resid > norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 > 52 KSP preconditioned resid norm 9.853979905043e-09 true resid > norm 9.853979905046e-09 ||Ae||/||Ax|| 1.684175984516e-03 > 0 KSP preconditioned resid norm 1.170745734096e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.170745734096e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 4.134603323396e-01 true resid > norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 > 1 KSP preconditioned resid norm 4.134603323396e-01 true resid > norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 > 53 KSP preconditioned resid norm 8.181596904133e-09 true resid > norm 8.181596904190e-09 ||Ae||/||Ax|| 1.398343527570e-03 > 0 KSP preconditioned resid norm 2.385292955090e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.385292955090e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.286979495932e-01 true resid > norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 > 1 KSP preconditioned resid norm 5.286979495932e-01 true resid > norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 > 54 KSP preconditioned resid norm 6.409203469073e-09 true resid > norm 6.409203469062e-09 ||Ae||/||Ax|| 1.095417959696e-03 > 0 KSP preconditioned resid norm 5.707419973376e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.707419973376e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.431212555078e-01 true resid > norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 > 1 KSP preconditioned resid norm 6.431212555078e-01 true resid > norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 > 55 KSP preconditioned resid norm 4.450638029919e-09 true resid > norm 4.450638029969e-09 ||Ae||/||Ax|| 7.606731247757e-04 > 0 KSP preconditioned resid norm 7.199725849178e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 7.199725849178e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.820447922491e-01 true resid > norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 > 1 KSP preconditioned resid norm 8.820447922491e-01 true resid > norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 > 56 KSP preconditioned resid norm 2.820717192827e-09 true resid > norm 2.820717192685e-09 ||Ae||/||Ax|| 4.820980153004e-04 > 0 KSP preconditioned resid norm 3.954316557881e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.954316557881e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.174082394688e+00 true resid > norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 > 1 KSP preconditioned resid norm 1.174082394688e+00 true resid > norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 > 57 KSP preconditioned resid norm 1.751635661806e-09 true resid > norm 1.751635661897e-09 ||Ae||/||Ax|| 2.993777888545e-04 > 0 KSP preconditioned resid norm 1.876208896020e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.876208896020e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.566896137837e+00 true resid > norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 > 1 KSP preconditioned resid norm 1.566896137837e+00 true resid > norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 > 58 KSP preconditioned resid norm 1.207891841472e-09 true resid > norm 1.207891841440e-09 ||Ae||/||Ax|| 2.064447513440e-04 > 0 KSP preconditioned resid norm 2.297261201903e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.297261201903e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.961454513785e+00 true resid > norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 > 1 KSP preconditioned resid norm 1.961454513785e+00 true resid > norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 > 59 KSP preconditioned resid norm 1.016539279879e-09 true resid > norm 1.016539279925e-09 ||Ae||/||Ax|| 1.737400582368e-04 > 0 KSP preconditioned resid norm 3.254668263088e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.254668263088e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.640547979130e+00 true resid > norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 > 1 KSP preconditioned resid norm 1.640547979130e+00 true resid > norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 > 60 KSP preconditioned resid norm 8.558459393246e-10 true resid > norm 8.558459393246e-10 ||Ae||/||Ax|| 1.462754330073e-04 > 0 KSP preconditioned resid norm 3.699188433787e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.699188433787e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 3.641405579973e+00 true resid > norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 > 1 KSP preconditioned resid norm 3.641405579973e+00 true resid > norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 > 61 KSP preconditioned resid norm 7.891395315495e-10 true resid > norm 7.891395315494e-10 ||Ae||/||Ax|| 1.348744223425e-04 > 0 KSP preconditioned resid norm 2.675881805310e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.675881805310e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.453713947810e+00 true resid > norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 > 1 KSP preconditioned resid norm 1.453713947810e+00 true resid > norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 > 62 KSP preconditioned resid norm 7.510552985484e-10 true resid > norm 7.510552985213e-10 ||Ae||/||Ax|| 1.283653213221e-04 > 0 KSP preconditioned resid norm 1.338840353148e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.338840353148e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.146220467265e+00 true resid > norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 > 1 KSP preconditioned resid norm 1.146220467265e+00 true resid > norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 > 63 KSP preconditioned resid norm 6.832740701908e-10 true resid > norm 6.832740702025e-10 ||Ae||/||Ax|| 1.167806095574e-04 > 0 KSP preconditioned resid norm 2.478031412327e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.478031412327e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.547561676965e+00 true resid > norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 > 1 KSP preconditioned resid norm 1.547561676965e+00 true resid > norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 > 64 KSP preconditioned resid norm 6.312416928849e-10 true resid > norm 6.312416928420e-10 ||Ae||/||Ax|| 1.078875854989e-04 > 0 KSP preconditioned resid norm 2.510304348429e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.510304348429e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.470709935588e+00 true resid > norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 > 1 KSP preconditioned resid norm 1.470709935588e+00 true resid > norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 > 65 KSP preconditioned resid norm 5.327354965414e-10 true resid > norm 5.327354965097e-10 ||Ae||/||Ax|| 9.105156880435e-05 > 0 KSP preconditioned resid norm 2.476662569003e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.476662569003e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.310145576275e+00 true resid > norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 > 1 KSP preconditioned resid norm 2.310145576275e+00 true resid > norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 > 66 KSP preconditioned resid norm 4.469580732178e-10 true resid > norm 4.469580732741e-10 ||Ae||/||Ax|| 7.639106841575e-05 > 0 KSP preconditioned resid norm 3.243520644840e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.243520644840e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.732155047053e+00 true resid > norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 > 1 KSP preconditioned resid norm 2.732155047053e+00 true resid > norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 > 67 KSP preconditioned resid norm 4.087445950732e-10 true resid > norm 4.087445950372e-10 ||Ae||/||Ax|| 6.985987767337e-05 > 0 KSP preconditioned resid norm 4.570320195732e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.570320195732e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.087601985141e+00 true resid > norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 > 1 KSP preconditioned resid norm 2.087601985141e+00 true resid > norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 > 68 KSP preconditioned resid norm 3.841153386650e-10 true resid > norm 3.841153386559e-10 ||Ae||/||Ax|| 6.565041078654e-05 > 0 KSP preconditioned resid norm 5.718860667569e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 5.718860667569e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.709934079219e+00 true resid > norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 > 1 KSP preconditioned resid norm 1.709934079219e+00 true resid > norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 > 69 KSP preconditioned resid norm 3.520264512665e-10 true resid > norm 3.520264512287e-10 ||Ae||/||Ax|| 6.016599392193e-05 > 0 KSP preconditioned resid norm 4.537109521149e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.537109521149e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.937349360261e+00 true resid > norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 > 1 KSP preconditioned resid norm 1.937349360261e+00 true resid > norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 > 70 KSP preconditioned resid norm 3.149991658898e-10 true resid > norm 3.149991658892e-10 ||Ae||/||Ax|| 5.383753929329e-05 > 0 KSP preconditioned resid norm 3.602956988994e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.602956988994e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.093057977304e+00 true resid > norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 > 1 KSP preconditioned resid norm 2.093057977304e+00 true resid > norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 > 71 KSP preconditioned resid norm 2.993160400510e-10 true resid > norm 2.993160400417e-10 ||Ae||/||Ax|| 5.115708488107e-05 > 0 KSP preconditioned resid norm 3.336754379842e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.336754379842e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.284284263997e+00 true resid > norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 > 1 KSP preconditioned resid norm 1.284284263997e+00 true resid > norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 > 72 KSP preconditioned resid norm 2.895591295761e-10 true resid > norm 2.895591295591e-10 ||Ae||/||Ax|| 4.948949934951e-05 > 0 KSP preconditioned resid norm 2.267067098558e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.267067098558e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.534973699953e-01 true resid > norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 > 1 KSP preconditioned resid norm 8.534973699953e-01 true resid > norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 > 73 KSP preconditioned resid norm 2.812425874590e-10 true resid > norm 2.812425874308e-10 ||Ae||/||Ax|| 4.806809189165e-05 > 0 KSP preconditioned resid norm 1.144552955812e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.144552955812e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.664517534844e-01 true resid > norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 > 1 KSP preconditioned resid norm 6.664517534844e-01 true resid > norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 > 74 KSP preconditioned resid norm 2.735484153434e-10 true resid > norm 2.735484153084e-10 ||Ae||/||Ax|| 4.675305573021e-05 > 0 KSP preconditioned resid norm 1.554702841568e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.554702841568e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.157380114110e-01 true resid > norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 > 1 KSP preconditioned resid norm 7.157380114110e-01 true resid > norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 > 75 KSP preconditioned resid norm 2.661574205602e-10 true resid > norm 2.661574205227e-10 ||Ae||/||Ax|| 4.548983660049e-05 > 0 KSP preconditioned resid norm 4.199331451693e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 4.199331451693e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 7.261245258089e-01 true resid > norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 > 1 KSP preconditioned resid norm 7.261245258089e-01 true resid > norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 > 76 KSP preconditioned resid norm 2.534998275631e-10 true resid > norm 2.534998275209e-10 ||Ae||/||Ax|| 4.332648591775e-05 > 0 KSP preconditioned resid norm 3.743395182510e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.743395182510e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.990417749799e-01 true resid > norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 > 1 KSP preconditioned resid norm 8.990417749799e-01 true resid > norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 > 77 KSP preconditioned resid norm 2.388049684536e-10 true resid > norm 2.388049684532e-10 ||Ae||/||Ax|| 4.081493941813e-05 > 0 KSP preconditioned resid norm 1.894155910280e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.894155910280e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.684190700742e-01 true resid > norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 > 1 KSP preconditioned resid norm 8.684190700742e-01 true resid > norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 > 78 KSP preconditioned resid norm 2.286512776556e-10 true resid > norm 2.286512776733e-10 ||Ae||/||Ax|| 3.907953886624e-05 > 0 KSP preconditioned resid norm 1.276207551837e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.276207551837e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 5.355779104233e-01 true resid > norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 > 1 KSP preconditioned resid norm 5.355779104233e-01 true resid > norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 > 79 KSP preconditioned resid norm 2.074225711250e-10 true resid > norm 2.074225711493e-10 ||Ae||/||Ax|| 3.545127109477e-05 > 0 KSP preconditioned resid norm 1.200062868567e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.200062868567e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 6.893976477109e-01 true resid > norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 > 1 KSP preconditioned resid norm 6.893976477109e-01 true resid > norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 > 80 KSP preconditioned resid norm 1.878908772984e-10 true resid > norm 1.878908773103e-10 ||Ae||/||Ax|| 3.211304532026e-05 > 0 KSP preconditioned resid norm 1.387129119694e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.387129119694e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.742548405622e-01 true resid > norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 > 1 KSP preconditioned resid norm 8.742548405622e-01 true resid > norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 > 81 KSP preconditioned resid norm 1.720957363240e-10 true resid > norm 1.720957363027e-10 ||Ae||/||Ax|| 2.941344603009e-05 > 0 KSP preconditioned resid norm 1.602115661552e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.602115661552e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.392172521059e-01 true resid > norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 > 1 KSP preconditioned resid norm 8.392172521059e-01 true resid > norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 > 82 KSP preconditioned resid norm 1.584755992063e-10 true resid > norm 1.584755991999e-10 ||Ae||/||Ax|| 2.708558378201e-05 > 0 KSP preconditioned resid norm 1.601197866084e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.601197866084e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 8.359479912386e-01 true resid > norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 > 1 KSP preconditioned resid norm 8.359479912386e-01 true resid > norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 > 83 KSP preconditioned resid norm 1.418702609113e-10 true resid > norm 1.418702608829e-10 ||Ae||/||Ax|| 2.424751101569e-05 > 0 KSP preconditioned resid norm 1.499465989768e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.499465989768e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.045754716297e+00 true resid > norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 > 1 KSP preconditioned resid norm 1.045754716297e+00 true resid > norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 > 84 KSP preconditioned resid norm 1.271329562402e-10 true resid > norm 1.271329562130e-10 ||Ae||/||Ax|| 2.172870999918e-05 > 0 KSP preconditioned resid norm 1.634503776014e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 1.634503776014e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.222338373864e+00 true resid > norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 > 1 KSP preconditioned resid norm 1.222338373864e+00 true resid > norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 > 85 KSP preconditioned resid norm 1.170830928689e-10 true resid > norm 1.170830928467e-10 ||Ae||/||Ax|| 2.001105493065e-05 > 0 KSP preconditioned resid norm 2.189478982761e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.189478982761e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.257863289529e+00 true resid > norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 > 1 KSP preconditioned resid norm 1.257863289529e+00 true resid > norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 > 86 KSP preconditioned resid norm 1.072794488701e-10 true resid > norm 1.072794488559e-10 ||Ae||/||Ax|| 1.833548202211e-05 > 0 KSP preconditioned resid norm 3.043239239438e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 3.043239239438e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 1.508125411857e+00 true resid > norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 > 1 KSP preconditioned resid norm 1.508125411857e+00 true resid > norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 > 87 KSP preconditioned resid norm 9.310419868753e-11 true resid > norm 9.310419867003e-11 ||Ae||/||Ax|| 1.591274357860e-05 > 0 KSP preconditioned resid norm 2.707043890291e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.707043890291e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.156895340838e+00 true resid > norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 > 1 KSP preconditioned resid norm 2.156895340838e+00 true resid > norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 > 88 KSP preconditioned resid norm 7.852841829285e-11 true resid > norm 7.852841828475e-11 ||Ae||/||Ax|| 1.342154920668e-05 > 0 KSP preconditioned resid norm 2.582289916941e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.582289916941e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.382288228383e+00 true resid > norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 > 1 KSP preconditioned resid norm 2.382288228383e+00 true resid > norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 > 89 KSP preconditioned resid norm 6.494099865988e-11 true resid > norm 6.494099869486e-11 ||Ae||/||Ax|| 1.109927881590e-05 > 0 KSP preconditioned resid norm 2.588940215193e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 0 KSP preconditioned resid norm 2.588940215193e+00 true resid > norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 2.412871891541e+00 true resid > norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 > 1 KSP preconditioned resid norm 2.412871891541e+00 true resid > norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 > 90 KSP preconditioned resid norm 5.586819688621e-11 true resid > norm 5.586819689526e-11 ||Ae||/||Ax|| 9.548616540312e-06 > 2 SNES Function norm 5.587280798316e-11 > SNES Object: > type: ls > line search variant: SNESLineSearchCubic > alpha=0.0001, maxstep=1e+08, minlambda=1e-12 > maximum iterations=50, maximum function evaluations=10000 > tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 > total number of linear solver iterations=107 > total number of function evaluations=3 > KSP Object: > type: fgmres > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > right preconditioning > PC Object: > type: mg > MG: type is FULL, levels=1 cycles=v > Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- > KSP Object:(mg_levels_0_) > type: gmres > GMRES: restart=30, using Classical (unmodified) Gram- > Schmidt Orthogonalization with no iterative refinement > GMRES: happy breakdown tolerance 1e-30 > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000 > left preconditioning > PC Object:(mg_levels_0_) > type: ilu > ILU: 0 levels of fill > ILU: factor fill ratio allocated 1 > ILU: tolerance for zero pivot 1e-12 > ILU: using diagonal shift to prevent zero pivot > ILU: using diagonal shift on blocks to prevent zero pivot > out-of-place factorization > matrix ordering: natural > ILU: factor fill ratio needed 1 > Factored matrix follows > Matrix Object: > type=seqaij, rows=39366, cols=39366 > package used to perform factorization: petsc > total: nonzeros=2125764, allocated nonzeros=2125764 > using I-node routines: found 19683 nodes, limit > used is 5 > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=39366, cols=39366 > total: nonzeros=2125764, allocated nonzeros=2125764 > using I-node routines: found 19683 nodes, limit used is 5 > linear system matrix = precond matrix: > Matrix Object: > type=seqaij, rows=39366, cols=39366 > total: nonzeros=2125764, allocated nonzeros=2125764 > using I-node routines: found 19683 nodes, limit used is 5 > > Thanks again, > Hal > > > > > Barry Smith wrote: >> Please run with >> -ksp_monitor_true_residual -snes_monitor - >> mg_levels_ksp_monitor_true_residua -mg_coarse_ksp_type gmres - >> mg_coarse_ksp_monitor_true_residual -snes_view >> and send the output for one time-step. >> Barry >> On Aug 26, 2009, at 2:34 PM, Hal Finkel wrote: >>> Hello, >>> >>> I've constructed a evolution scheme for a time-dependent 3D PDE >>> system using the DMMG layer. The core of the code is essentially a >>> loop around DMMGSolve in the spirit of snes/examples/tutorials/ >>> ex29.c. If I only run with one level, then the code runs and >>> produces sensible results. Passing the option - >>> mg_levels_0_ksp_type bcgsl increases the solver speed by about >>> 30%, and -mg_levels_0_pc_type fieldsplit gives me another 5% >>> improvement. Does that say something about the structure of my >>> problem? >>> >>> If I run with two levels, the coarse solve seems to work, but the >>> fine grid's KSP never seems to complete. If I run with - >>> mg_levels_1_ksp_monitor_true_residual, I can see the residual does >>> not seem to be converging at all: >>> >>> 0 KSP preconditioned resid norm 1.979915874387e+02 true resid >>> norm 1.889855551272e+04 ||Ae||/||Ax|| 1.889855551272e+04 >>> 0 KSP preconditioned resid norm 3.833279709488e+06 true resid >>> norm 3.665290817713e+08 ||Ae||/||Ax|| 3.665290817713e+08 >>> 0 KSP preconditioned resid norm 5.080465654647e+01 true resid >>> norm 8.284057252719e+03 ||Ae||/||Ax|| 8.284057252719e+03 >>> 1 KSP preconditioned resid norm 6.057649668915e+00 true resid >>> norm 5.136009463814e+02 ||Ae||/||Ax|| 5.136009463814e+02 >>> 0 KSP preconditioned resid norm 5.511785811223e+04 true resid >>> norm 1.055162492334e+07 ||Ae||/||Ax|| 1.055162492334e+07 >>> 0 KSP preconditioned resid norm 2.164672340919e+01 true resid >>> norm 2.999406329054e+04 ||Ae||/||Ax|| 2.999406329054e+04 >>> 0 KSP preconditioned resid norm 5.275983407134e+05 true resid >>> norm 9.196351032040e+08 ||Ae||/||Ax|| 9.196351032040e+08 >>> 0 KSP preconditioned resid norm 3.338344860439e+01 true resid >>> norm 5.133553722923e+03 ||Ae||/||Ax|| 5.133553722923e+03 >>> 1 KSP preconditioned resid norm 4.306407436023e+00 true resid >>> norm 3.361745947223e+02 ||Ae||/||Ax|| 3.361745947223e+02 >>> ... >>> >>> If I run with -pc_mg_monitor I see, after the course solve, only >>> things like: >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 1.979915874387e+02 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 3.833279709488e+06 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 5.080465654647e+01 >>> 1 KSP Residual norm 6.057649668915e+00 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 5.511785811223e+04 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 2.164672340919e+01 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 5.275983407134e+05 >>> Residual norms for mg_levels_1_ solve. >>> 0 KSP Residual norm 3.338344860439e+01 >>> 1 KSP Residual norm 4.306407436023e+00 >>> ... >>> >>> running with -info shows: >>> [0] KSPDefaultConverged(): user has provided nonzero initial >>> guess, computing 2-norm of preconditioned RHS >>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial >>> right hand size norm 0.0066162, current residual norm 23282.7 at >>> iteration 0 >>> [0] KSPDefaultConverged(): user has provided nonzero initial >>> guess, computing 2-norm of preconditioned RHS >>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial >>> right hand size norm 0.00287711, current residual norm 38.2205 at >>> iteration 0 >>> [0] KSPDefaultConverged(): user has provided nonzero initial >>> guess, computing 2-norm of preconditioned RHS >>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial >>> right hand size norm 0.00287711, current residual norm 809191 at >>> iteration 0 >>> [0] KSPDefaultConverged(): user has provided nonzero initial >>> guess, computing 2-norm of preconditioned RHS >>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial >>> right hand size norm 0.00970186, current residual norm 186.983 at >>> iteration 0 >>> [0] KSPDefaultConverged(): user has provided nonzero initial >>> guess, computing 2-norm of preconditioned RHS >>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial >>> right hand size norm 0.00970186, current residual norm 3.62292e+06 >>> at iteration 0 >>> >>> What is a good way to debug this? >>> >>> No matter how I run, I've arranged the code so that the fine grid >>> is *always* the same size; it has to be because the initial >>> conditions are loaded from a file. >>> >>> In general, however, I don't understand exactly what the mg >>> preconditioner is doing. Is there a writeup somewhere which is >>> more verbose than the manual? In the one-level case, how are the >>> results of the solve used to precondition the Jacobian? >>> >>> In an 1-processor, 1-level case, running with -snes_view gives: >>> SNES Object: >>> type: ls >>> line search variant: SNESLineSearchCubic >>> alpha=0.0001, maxstep=1e+08, minlambda=1e-12 >>> maximum iterations=50, maximum function evaluations=10000 >>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>> total number of linear solver iterations=36 >>> total number of function evaluations=3 >>> KSP Object: >>> type: fgmres >>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >>> Orthogonalization with no iterative refinement >>> GMRES: happy breakdown tolerance 1e-30 >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> right preconditioning >>> PC Object: >>> type: mg >>> MG: type is FULL, levels=1 cycles=v >>> Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- >>> KSP Object:(mg_levels_0_) >>> type: bcgsl >>> BCGSL: Ell = 2 >>> BCGSL: Delta = 0 >>> maximum iterations=1, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>> left preconditioning >>> PC Object:(mg_levels_0_) >>> type: fieldsplit >>> FieldSplit with MULTIPLICATIVE composition: total splits = >>> 2, blocksize = 2 >>> Solver info for each split is in the following KSP objects: >>> Split number 0 Fields 0 >>> KSP Object:(mg_levels_0_fieldsplit_0_) >>> type: preonly >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, >>> divergence=10000 >>> left preconditioning >>> PC Object:(mg_levels_0_fieldsplit_0_) >>> type: ilu >>> ILU: 0 levels of fill >>> ILU: factor fill ratio allocated 1 >>> ILU: tolerance for zero pivot 1e-12 >>> ILU: using diagonal shift to prevent zero pivot >>> ILU: using diagonal shift on blocks to prevent zero >>> pivot >>> out-of-place factorization >>> matrix ordering: natural >>> ILU: factor fill ratio needed 1 >>> Factored matrix follows >>> Matrix Object: >>> type=seqaij, rows=19683, cols=19683 >>> package used to perform factorization: petsc >>> total: nonzeros=531441, allocated nonzeros=531441 >>> not using I-node routines >>> linear system matrix = precond matrix: >>> Matrix Object: >>> type=seqaij, rows=19683, cols=19683 >>> total: nonzeros=531441, allocated nonzeros=531441 >>> not using I-node routines >>> Split number 1 Fields 1 >>> KSP Object:(mg_levels_0_fieldsplit_1_) >>> type: preonly >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, >>> divergence=10000 >>> left preconditioning >>> PC Object:(mg_levels_0_fieldsplit_1_) >>> type: ilu >>> ILU: 0 levels of fill >>> ILU: factor fill ratio allocated 1 >>> ILU: tolerance for zero pivot 1e-12 >>> ILU: using diagonal shift to prevent zero pivot >>> ILU: using diagonal shift on blocks to prevent zero >>> pivot >>> out-of-place factorization >>> matrix ordering: natural >>> ILU: factor fill ratio needed 1 >>> Factored matrix follows >>> Matrix Object: >>> type=seqaij, rows=19683, cols=19683 >>> package used to perform factorization: petsc >>> total: nonzeros=531441, allocated nonzeros=531441 >>> not using I-node routines >>> linear system matrix = precond matrix: >>> Matrix Object: >>> type=seqaij, rows=19683, cols=19683 >>> total: nonzeros=531441, allocated nonzeros=531441 >>> not using I-node routines >>> linear system matrix = precond matrix: >>> Matrix Object: >>> type=seqaij, rows=39366, cols=39366 >>> total: nonzeros=2125764, allocated nonzeros=2125764 >>> using I-node routines: found 19683 nodes, limit used is 5 >>> linear system matrix = precond matrix: >>> Matrix Object: >>> type=seqaij, rows=39366, cols=39366 >>> total: nonzeros=2125764, allocated nonzeros=2125764 >>> using I-node routines: found 19683 nodes, limit used is 5 >>> >>> Thank you in advance, >>> Hal From hal.finkel at yale.edu Wed Aug 26 16:11:51 2009 From: hal.finkel at yale.edu (Hal Finkel) Date: Wed, 26 Aug 2009 17:11:51 -0400 Subject: Multigrid Questions In-Reply-To: <1570E834-D71E-4F0B-B4C4-90C926A2988D@mcs.anl.gov> References: <4A958E3C.2070400@yale.edu> <2EF3551E-FEA7-4A09-BE94-41BA88E0190B@mcs.anl.gov> <4A959F07.9010506@yale.edu> <1570E834-D71E-4F0B-B4C4-90C926A2988D@mcs.anl.gov> Message-ID: <4A95A517.6020506@yale.edu> Barry, Because the solve never completes, you never see the -snes_view output. Here is the 2-level case. With -dmmg_grid_sequence (you actually do see the -snes_view output this time, so maybe without this flag even the coarse solve never finishes?): 0 SNES Function norm 5.473992351804e+03 0 KSP preconditioned resid norm 2.533502558402e+01 true resid norm 5.473992351804e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.533502558402e+01 true resid norm 5.473992351804e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.968743708214e-11 true resid norm 1.324235124537e-11 ||Ae||/||Ax|| 2.419139522729e-15 1 KSP preconditioned resid norm 1.968743708214e-11 true resid norm 1.324235124537e-11 ||Ae||/||Ax|| 2.419139522729e-15 1 SNES Function norm 1.624916397041e-01 0 KSP preconditioned resid norm 2.944703982502e-04 true resid norm 1.624916397041e-01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.944703982502e-04 true resid norm 1.624916397041e-01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.360780048165e-06 true resid norm 1.281855538252e-03 ||Ae||/||Ax|| 7.888747633945e-03 1 KSP preconditioned resid norm 3.360780048165e-06 true resid norm 1.281855538252e-03 ||Ae||/||Ax|| 7.888747633945e-03 2 SNES Function norm 1.281855487791e-03 0 KSP preconditioned resid norm 3.360764482735e-06 true resid norm 1.281855487791e-03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.360764482735e-06 true resid norm 1.281855487791e-03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.416556005968e-08 true resid norm 2.978313158215e-05 ||Ae||/||Ax|| 2.323439097919e-02 1 KSP preconditioned resid norm 5.416556005968e-08 true resid norm 2.978313158215e-05 ||Ae||/||Ax|| 2.323439097919e-02 3 SNES Function norm 2.978313156388e-05 SNES Object: type: ls line search variant: SNESLineSearchCubic alpha=0.0001, maxstep=1e+08, minlambda=1e-12 maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=3 total number of function evaluations=4 KSP Object: type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 right preconditioning PC Object: type: mg MG: type is FULL, levels=1 cycles=v Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- KSP Object:(mg_levels_0_) type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=1, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object:(mg_levels_0_) type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: factor fill ratio needed 10.3934 Factored matrix follows Matrix Object: type=seqaij, rows=1458, cols=1458 package used to perform factorization: petsc total: nonzeros=818292, allocated nonzeros=818292 using I-node routines: found 640 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=1458, cols=1458 total: nonzeros=78732, allocated nonzeros=78732 using I-node routines: found 729 nodes, limit used is 5 linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=1458, cols=1458 total: nonzeros=78732, allocated nonzeros=78732 using I-node routines: found 729 nodes, limit used is 5 0 SNES Function norm 1.308698287351e+00 0 KSP preconditioned resid norm 1.308698287351e+00 true resid norm 1.308698287351e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.671892130808e-01 true resid norm 1.866582511975e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.671892130808e-01 true resid norm 1.866582511975e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.246743442682e-13 true resid norm 1.380198967143e-12 ||Ae||/||Ax|| 7.394256392572e-13 1 KSP preconditioned resid norm 1.246743442682e-13 true resid norm 1.380198967143e-12 ||Ae||/||Ax|| 7.394256392572e-13 0 KSP preconditioned resid norm 4.951265543603e-01 true resid norm 1.017496416000e+03 ||Ae||/||Ax|| 1.017496416000e+03 1 KSP preconditioned resid norm 1.741881317500e-01 true resid norm 2.697856264290e+02 ||Ae||/||Ax|| 2.697856264290e+02 0 KSP preconditioned resid norm 1.855265388514e+03 true resid norm 4.457571496212e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.855265388514e+03 true resid norm 4.457571496212e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.522733286227e-10 true resid norm 6.897121799185e-10 ||Ae||/||Ax|| 1.547282372262e-12 1 KSP preconditioned resid norm 5.522733286227e-10 true resid norm 6.897121799185e-10 ||Ae||/||Ax|| 1.547282372262e-12 0 KSP preconditioned resid norm 3.139459156768e+03 true resid norm 6.178203349663e+06 ||Ae||/||Ax|| 6.178203349663e+06 1 KSP preconditioned resid norm 1.307366779482e+00 true resid norm 1.307366779482e+00 ||Ae||/||Ax|| 9.989825707873e-01 0 KSP preconditioned resid norm 9.126107408200e+00 true resid norm 1.381216040059e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.126107408200e+00 true resid norm 1.381216040059e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.550817052315e-12 true resid norm 3.042123300613e-12 ||Ae||/||Ax|| 2.202496360008e-12 1 KSP preconditioned resid norm 2.550817052315e-12 true resid norm 3.042123300613e-12 ||Ae||/||Ax|| 2.202496360008e-12 0 KSP preconditioned resid norm 1.542594768751e+01 true resid norm 3.049460348051e+04 ||Ae||/||Ax|| 3.049460348051e+04 0 KSP preconditioned resid norm 2.793298665113e+05 true resid norm 4.194826255177e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.793298665113e+05 true resid norm 4.194826255177e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.370808612517e-08 true resid norm 9.579338218581e-08 ||Ae||/||Ax|| 2.283607862604e-12 1 KSP preconditioned resid norm 8.370808612517e-08 true resid norm 9.579338218581e-08 ||Ae||/||Ax|| 2.283607862604e-12 0 KSP preconditioned resid norm 4.724281628181e+05 true resid norm 9.357252729253e+08 ||Ae||/||Ax|| 9.357252729253e+08 2 KSP preconditioned resid norm 1.286291199119e+00 true resid norm 1.286291199119e+00 ||Ae||/||Ax|| 9.828783391491e-01 0 KSP preconditioned resid norm 4.853086257912e+00 true resid norm 1.762407227405e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.853086257912e+00 true resid norm 1.762407227405e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.188660535877e-12 true resid norm 1.095897917516e-12 ||Ae||/||Ax|| 6.218187831250e-13 1 KSP preconditioned resid norm 1.188660535877e-12 true resid norm 1.095897917516e-12 ||Ae||/||Ax|| 6.218187831250e-13 0 KSP preconditioned resid norm 7.354250474791e+00 true resid norm 5.806918120227e+03 ||Ae||/||Ax|| 5.806918120227e+03 0 KSP preconditioned resid norm 9.507342745042e+04 true resid norm 9.101379563722e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.507342745042e+04 true resid norm 9.101379563722e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.507856901997e-08 true resid norm 1.203978211327e-08 ||Ae||/||Ax|| 1.322852434510e-12 1 KSP preconditioned resid norm 2.507856901997e-08 true resid norm 1.203978211327e-08 ||Ae||/||Ax|| 1.322852434510e-12 0 KSP preconditioned resid norm 1.424177612658e+05 true resid norm 1.080775490793e+08 ||Ae||/||Ax|| 1.080775490793e+08 3 KSP preconditioned resid norm 1.281076066870e+00 true resid norm 1.281076066870e+00 ||Ae||/||Ax|| 9.788933623990e-01 0 KSP preconditioned resid norm 1.031612292550e+02 true resid norm 1.605474040138e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.031612292550e+02 true resid norm 1.605474040138e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.422227657515e-11 true resid norm 1.826368840914e-12 ||Ae||/||Ax|| 1.137588522302e-12 1 KSP preconditioned resid norm 2.422227657515e-11 true resid norm 1.826368840914e-12 ||Ae||/||Ax|| 1.137588522302e-12 0 KSP preconditioned resid norm 1.516245422282e+02 true resid norm 1.511985350228e+04 ||Ae||/||Ax|| 1.511985350228e+04 0 KSP preconditioned resid norm 1.998524571466e+06 true resid norm 2.114287680254e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.998524571466e+06 true resid norm 2.114287680254e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.919571300159e-07 true resid norm 3.379476213156e-08 ||Ae||/||Ax|| 1.598399425356e-12 1 KSP preconditioned resid norm 4.919571300159e-07 true resid norm 3.379476213156e-08 ||Ae||/||Ax|| 1.598399425356e-12 0 KSP preconditioned resid norm 2.937372721762e+06 true resid norm 2.886115000780e+08 ||Ae||/||Ax|| 2.886115000780e+08 4 KSP preconditioned resid norm 1.274608512034e+00 true resid norm 1.274608512034e+00 ||Ae||/||Ax|| 9.739513869267e-01 0 KSP preconditioned resid norm 8.584999416955e+01 true resid norm 1.676010194851e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.584999416955e+01 true resid norm 1.676010194851e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.121159984852e-11 true resid norm 1.647263991685e-12 ||Ae||/||Ax|| 9.828484317967e-13 1 KSP preconditioned resid norm 2.121159984852e-11 true resid norm 1.647263991685e-12 ||Ae||/||Ax|| 9.828484317967e-13 0 KSP preconditioned resid norm 1.261867401997e+02 true resid norm 1.282997762347e+04 ||Ae||/||Ax|| 1.282997762347e+04 0 KSP preconditioned resid norm 1.663127865840e+06 true resid norm 1.809949324252e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.663127865840e+06 true resid norm 1.809949324252e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.715693256069e-07 true resid norm 2.797939280490e-08 ||Ae||/||Ax|| 1.545866087520e-12 1 KSP preconditioned resid norm 3.715693256069e-07 true resid norm 2.797939280490e-08 ||Ae||/||Ax|| 1.545866087520e-12 0 KSP preconditioned resid norm 2.444432338934e+06 true resid norm 2.407335970627e+08 ||Ae||/||Ax|| 2.407335970627e+08 5 KSP preconditioned resid norm 1.269640505517e+00 true resid norm 1.269640505517e+00 ||Ae||/||Ax|| 9.701552434119e-01 0 KSP preconditioned resid norm 8.500173536946e+00 true resid norm 1.474437012468e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.500173536946e+00 true resid norm 1.474437012468e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.731992835726e-12 true resid norm 3.130508679725e-12 ||Ae||/||Ax|| 2.123189158474e-12 1 KSP preconditioned resid norm 2.731992835726e-12 true resid norm 3.130508679725e-12 ||Ae||/||Ax|| 2.123189158474e-12 0 KSP preconditioned resid norm 1.445317443628e+01 true resid norm 2.836860714321e+04 ||Ae||/||Ax|| 2.836860714321e+04 0 KSP preconditioned resid norm 2.598079909146e+05 true resid norm 3.904701021330e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.598079909146e+05 true resid norm 3.904701021330e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.712159042515e-08 true resid norm 9.430555275347e-08 ||Ae||/||Ax|| 2.415179862384e-12 1 KSP preconditioned resid norm 7.712159042515e-08 true resid norm 9.430555275347e-08 ||Ae||/||Ax|| 2.415179862384e-12 0 KSP preconditioned resid norm 4.422339903409e+05 true resid norm 8.699841646170e+08 ||Ae||/||Ax|| 8.699841646170e+08 6 KSP preconditioned resid norm 1.240293731529e+00 true resid norm 1.240293731529e+00 ||Ae||/||Ax|| 9.477308433251e-01 0 KSP preconditioned resid norm 3.817329358402e+00 true resid norm 1.878328844937e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.817329358402e+00 true resid norm 1.878328844937e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.336488310399e-12 true resid norm 1.968321178329e-12 ||Ae||/||Ax|| 1.047910850986e-12 1 KSP preconditioned resid norm 1.336488310399e-12 true resid norm 1.968321178329e-12 ||Ae||/||Ax|| 1.047910850986e-12 0 KSP preconditioned resid norm 6.496038663865e+00 true resid norm 1.203945386556e+04 ||Ae||/||Ax|| 1.203945386556e+04 0 KSP preconditioned resid norm 1.101956935839e+05 true resid norm 1.690737730774e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.101956935839e+05 true resid norm 1.690737730774e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.505687476217e-08 true resid norm 3.868649531567e-08 ||Ae||/||Ax|| 2.288142898306e-12 1 KSP preconditioned resid norm 3.505687476217e-08 true resid norm 3.868649531567e-08 ||Ae||/||Ax|| 2.288142898306e-12 0 KSP preconditioned resid norm 1.866393633809e+05 true resid norm 3.607809698759e+08 ||Ae||/||Ax|| 3.607809698759e+08 7 KSP preconditioned resid norm 1.192185750133e+00 true resid norm 1.192185750133e+00 ||Ae||/||Ax|| 9.109706657799e-01 0 KSP preconditioned resid norm 1.103198681366e+00 true resid norm 2.016429497095e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.103198681366e+00 true resid norm 2.016429497095e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.260860606235e-13 true resid norm 1.871444107996e-13 ||Ae||/||Ax|| 9.280979626076e-14 1 KSP preconditioned resid norm 2.260860606235e-13 true resid norm 1.871444107996e-13 ||Ae||/||Ax|| 9.280979626076e-14 0 KSP preconditioned resid norm 1.997708017818e+00 true resid norm 2.122037753565e+03 ||Ae||/||Ax|| 2.122037753565e+03 1 KSP preconditioned resid norm 4.618193595607e-01 true resid norm 2.540609128927e+02 ||Ae||/||Ax|| 2.540609128927e+02 0 KSP preconditioned resid norm 5.299709888224e+02 true resid norm 4.801745769741e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.299709888224e+02 true resid norm 4.801745769741e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.017584542111e-10 true resid norm 5.940522720915e-11 ||Ae||/||Ax|| 1.237158942973e-13 1 KSP preconditioned resid norm 1.017584542111e-10 true resid norm 5.940522720915e-11 ||Ae||/||Ax|| 1.237158942973e-13 0 KSP preconditioned resid norm 9.416500710791e+02 true resid norm 8.177676944591e+05 ||Ae||/||Ax|| 8.177676944591e+05 8 KSP preconditioned resid norm 1.187310607316e+00 true resid norm 1.187310607316e+00 ||Ae||/||Ax|| 9.072454810953e-01 0 KSP preconditioned resid norm 1.191721427949e+01 true resid norm 1.804358617749e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.191721427949e+01 true resid norm 1.804358617749e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.618507637154e-12 true resid norm 2.010483783173e-12 ||Ae||/||Ax|| 1.114237360243e-12 1 KSP preconditioned resid norm 3.618507637154e-12 true resid norm 2.010483783173e-12 ||Ae||/||Ax|| 1.114237360243e-12 0 KSP preconditioned resid norm 1.957620812169e+01 true resid norm 1.741562232256e+04 ||Ae||/||Ax|| 1.741562232256e+04 1 KSP preconditioned resid norm 3.235571300995e+00 true resid norm 1.560872557301e+03 ||Ae||/||Ax|| 1.560872557301e+03 0 KSP preconditioned resid norm 2.146086637228e+04 true resid norm 2.116725369143e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.146086637228e+04 true resid norm 2.116725369143e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.012680009150e-09 true resid norm 4.641272711834e-09 ||Ae||/||Ax|| 2.192666455221e-12 1 KSP preconditioned resid norm 5.012680009150e-09 true resid norm 4.641272711834e-09 ||Ae||/||Ax|| 2.192666455221e-12 0 KSP preconditioned resid norm 3.553939957626e+04 true resid norm 4.552924260777e+07 ||Ae||/||Ax|| 4.552924260777e+07 9 KSP preconditioned resid norm 1.184881170018e+00 true resid norm 1.184881170018e+00 ||Ae||/||Ax|| 9.053891041734e-01 0 KSP preconditioned resid norm 9.013294569631e+00 true resid norm 1.590917299780e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.013294569631e+00 true resid norm 1.590917299780e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.522672180871e-12 true resid norm 2.646584400435e-12 ||Ae||/||Ax|| 1.663558753683e-12 1 KSP preconditioned resid norm 3.522672180871e-12 true resid norm 2.646584400435e-12 ||Ae||/||Ax|| 1.663558753683e-12 0 KSP preconditioned resid norm 1.609904368624e+01 true resid norm 2.548311040691e+04 ||Ae||/||Ax|| 2.548311040691e+04 1 KSP preconditioned resid norm 3.138102119429e+00 true resid norm 3.221616601602e+03 ||Ae||/||Ax|| 3.221616601602e+03 0 KSP preconditioned resid norm 2.966723868503e+04 true resid norm 4.385489957291e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.966723868503e+04 true resid norm 4.385489957291e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.554991804882e-09 true resid norm 9.729410050451e-09 ||Ae||/||Ax|| 2.218545737239e-12 1 KSP preconditioned resid norm 9.554991804882e-09 true resid norm 9.729410050451e-09 ||Ae||/||Ax|| 2.218545737239e-12 0 KSP preconditioned resid norm 5.467848851613e+04 true resid norm 9.720762904183e+07 ||Ae||/||Ax|| 9.720762904183e+07 10 KSP preconditioned resid norm 1.183250603772e+00 true resid norm 1.183250603772e+00 ||Ae||/||Ax|| 9.041431590527e-01 0 KSP preconditioned resid norm 1.139917359003e+02 true resid norm 1.461985081857e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.139917359003e+02 true resid norm 1.461985081857e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.936778552378e-11 true resid norm 2.155858862569e-12 ||Ae||/||Ax|| 1.474610712053e-12 1 KSP preconditioned resid norm 1.936778552378e-11 true resid norm 2.155858862569e-12 ||Ae||/||Ax|| 1.474610712053e-12 0 KSP preconditioned resid norm 1.699009818121e+02 true resid norm 1.966564654661e+04 ||Ae||/||Ax|| 1.966564654661e+04 0 KSP preconditioned resid norm 2.209481632608e+06 true resid norm 2.722888057718e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.209481632608e+06 true resid norm 2.722888057718e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.439021080164e-07 true resid norm 5.024351882292e-08 ||Ae||/||Ax|| 1.845228953886e-12 1 KSP preconditioned resid norm 3.439021080164e-07 true resid norm 5.024351882292e-08 ||Ae||/||Ax|| 1.845228953886e-12 0 KSP preconditioned resid norm 3.293067141092e+06 true resid norm 4.566021924053e+08 ||Ae||/||Ax|| 4.566021924053e+08 11 KSP preconditioned resid norm 1.179842952379e+00 true resid norm 1.179842952379e+00 ||Ae||/||Ax|| 9.015393110718e-01 0 KSP preconditioned resid norm 9.312336333634e+00 true resid norm 1.536038920080e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.312336333634e+00 true resid norm 1.536038920080e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.543499555963e-12 true resid norm 2.831415628740e-12 ||Ae||/||Ax|| 1.843322842752e-12 1 KSP preconditioned resid norm 3.543499555963e-12 true resid norm 2.831415628740e-12 ||Ae||/||Ax|| 1.843322842752e-12 0 KSP preconditioned resid norm 1.627124439842e+01 true resid norm 2.454100587432e+04 ||Ae||/||Ax|| 2.454100587432e+04 0 KSP preconditioned resid norm 2.438191586954e+05 true resid norm 3.383001147319e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.438191586954e+05 true resid norm 3.383001147319e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.198980589988e-08 true resid norm 8.440942648826e-08 ||Ae||/||Ax|| 2.495104873232e-12 1 KSP preconditioned resid norm 7.198980589988e-08 true resid norm 8.440942648826e-08 ||Ae||/||Ax|| 2.495104873232e-12 0 KSP preconditioned resid norm 4.383111884338e+05 true resid norm 7.484515504412e+08 ||Ae||/||Ax|| 7.484515504412e+08 12 KSP preconditioned resid norm 1.175376162628e+00 true resid norm 1.175376162628e+00 ||Ae||/||Ax|| 8.981261563404e-01 0 KSP preconditioned resid norm 7.051008824068e+01 true resid norm 1.585083538737e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.051008824068e+01 true resid norm 1.585083538737e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.165012706118e-11 true resid norm 2.206772755388e-12 ||Ae||/||Ax|| 1.392212272387e-12 1 KSP preconditioned resid norm 1.165012706118e-11 true resid norm 2.206772755388e-12 ||Ae||/||Ax|| 1.392212272387e-12 0 KSP preconditioned resid norm 1.050376935802e+02 true resid norm 1.866739366731e+04 ||Ae||/||Ax|| 1.866739366731e+04 0 KSP preconditioned resid norm 1.368703621692e+06 true resid norm 2.585360901893e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.368703621692e+06 true resid norm 2.585360901893e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.169639697438e-07 true resid norm 5.809353183331e-08 ||Ae||/||Ax|| 2.247018270864e-12 1 KSP preconditioned resid norm 2.169639697438e-07 true resid norm 5.809353183331e-08 ||Ae||/||Ax|| 2.247018270864e-12 0 KSP preconditioned resid norm 2.043969827711e+06 true resid norm 5.113529582468e+08 ||Ae||/||Ax|| 5.113529582468e+08 13 KSP preconditioned resid norm 1.163992595272e+00 true resid norm 1.163992595272e+00 ||Ae||/||Ax|| 8.894277669060e-01 0 KSP preconditioned resid norm 8.515303915212e+00 true resid norm 1.904882460913e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.515303915212e+00 true resid norm 1.904882460913e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.570331341241e-12 true resid norm 3.322159253556e-13 ||Ae||/||Ax|| 1.744023225435e-13 1 KSP preconditioned resid norm 1.570331341241e-12 true resid norm 3.322159253556e-13 ||Ae||/||Ax|| 1.744023225435e-13 0 KSP preconditioned resid norm 1.443576158217e+01 true resid norm 3.491626175321e+03 ||Ae||/||Ax|| 3.491626175321e+03 1 KSP preconditioned resid norm 2.144436548969e+00 true resid norm 2.584436198878e+02 ||Ae||/||Ax|| 2.584436198878e+02 0 KSP preconditioned resid norm 6.082095393362e+03 true resid norm 3.982679317553e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.082095393362e+03 true resid norm 3.982679317553e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.609299091694e-09 true resid norm 2.053609341656e-10 ||Ae||/||Ax|| 5.156351234720e-13 1 KSP preconditioned resid norm 1.609299091694e-09 true resid norm 2.053609341656e-10 ||Ae||/||Ax|| 5.156351234720e-13 0 KSP preconditioned resid norm 9.468237802274e+03 true resid norm 1.500791057830e+06 ||Ae||/||Ax|| 1.500791057830e+06 14 KSP preconditioned resid norm 1.161954853862e+00 true resid norm 1.161954853862e+00 ||Ae||/||Ax|| 8.878706918874e-01 0 KSP preconditioned resid norm 7.268493138378e+01 true resid norm 1.658381589506e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.268493138378e+01 true resid norm 1.658381589506e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.236800083833e-11 true resid norm 1.609673637112e-12 ||Ae||/||Ax|| 9.706292250817e-13 1 KSP preconditioned resid norm 1.236800083833e-11 true resid norm 1.609673637112e-12 ||Ae||/||Ax|| 9.706292250817e-13 0 KSP preconditioned resid norm 1.090962150823e+02 true resid norm 1.422422881859e+04 ||Ae||/||Ax|| 1.422422881859e+04 0 KSP preconditioned resid norm 1.392543317987e+06 true resid norm 1.985413188675e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.392543317987e+06 true resid norm 1.985413188675e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.159467768755e-07 true resid norm 3.770676244274e-08 ||Ae||/||Ax|| 1.899189682924e-12 1 KSP preconditioned resid norm 2.159467768755e-07 true resid norm 3.770676244274e-08 ||Ae||/||Ax|| 1.899189682924e-12 0 KSP preconditioned resid norm 2.078324384653e+06 true resid norm 3.490817405338e+08 ||Ae||/||Ax|| 3.490817405338e+08 15 KSP preconditioned resid norm 1.161122807373e+00 true resid norm 1.161122807373e+00 ||Ae||/||Ax|| 8.872349101363e-01 0 KSP preconditioned resid norm 4.486384106202e+01 true resid norm 1.505866250635e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.486384106202e+01 true resid norm 1.505866250635e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.107455382253e-12 true resid norm 2.713186111845e-12 ||Ae||/||Ax|| 1.801744418337e-12 1 KSP preconditioned resid norm 7.107455382253e-12 true resid norm 2.713186111845e-12 ||Ae||/||Ax|| 1.801744418337e-12 0 KSP preconditioned resid norm 6.731100076983e+01 true resid norm 2.507166440335e+04 ||Ae||/||Ax|| 2.507166440335e+04 0 KSP preconditioned resid norm 8.735838913666e+05 true resid norm 3.455516905875e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.735838913666e+05 true resid norm 3.455516905875e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.763472347693e-07 true resid norm 8.184396405326e-08 ||Ae||/||Ax|| 2.368501335187e-12 1 KSP preconditioned resid norm 1.763472347693e-07 true resid norm 8.184396405326e-08 ||Ae||/||Ax|| 2.368501335187e-12 0 KSP preconditioned resid norm 1.318309422898e+06 true resid norm 7.524118563334e+08 ||Ae||/||Ax|| 7.524118563334e+08 16 KSP preconditioned resid norm 1.160759375595e+00 true resid norm 1.160759375595e+00 ||Ae||/||Ax|| 8.869572053499e-01 0 KSP preconditioned resid norm 1.049374964108e+02 true resid norm 1.434877879459e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.049374964108e+02 true resid norm 1.434877879459e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.462232523302e-11 true resid norm 2.449064800648e-12 ||Ae||/||Ax|| 1.706810618317e-12 1 KSP preconditioned resid norm 1.462232523302e-11 true resid norm 2.449064800648e-12 ||Ae||/||Ax|| 1.706810618317e-12 0 KSP preconditioned resid norm 1.563689176972e+02 true resid norm 2.234748831325e+04 ||Ae||/||Ax|| 2.234748831325e+04 0 KSP preconditioned resid norm 2.034170341786e+06 true resid norm 3.086532544601e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.034170341786e+06 true resid norm 3.086532544601e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.973509809548e-07 true resid norm 6.238722918085e-08 ||Ae||/||Ax|| 2.021272359171e-12 1 KSP preconditioned resid norm 2.973509809548e-07 true resid norm 6.238722918085e-08 ||Ae||/||Ax|| 2.021272359171e-12 0 KSP preconditioned resid norm 3.034308733052e+06 true resid norm 5.829656771747e+08 ||Ae||/||Ax|| 5.829656771747e+08 17 KSP preconditioned resid norm 1.159720744650e+00 true resid norm 1.159720744650e+00 ||Ae||/||Ax|| 8.861635686847e-01 0 KSP preconditioned resid norm 6.945410558826e+00 true resid norm 1.698126116149e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.945410558826e+00 true resid norm 1.698126116149e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.046608765850e-12 true resid norm 5.634458822597e-13 ||Ae||/||Ax|| 3.318044972639e-13 1 KSP preconditioned resid norm 2.046608765850e-12 true resid norm 5.634458822597e-13 ||Ae||/||Ax|| 3.318044972639e-13 0 KSP preconditioned resid norm 1.177325940074e+01 true resid norm 5.082427013645e+03 ||Ae||/||Ax|| 5.082427013645e+03 1 KSP preconditioned resid norm 2.205234596357e+00 true resid norm 4.860264071810e+02 ||Ae||/||Ax|| 4.860264071810e+02 0 KSP preconditioned resid norm 1.073978588334e+04 true resid norm 7.396119503065e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.073978588334e+04 true resid norm 7.396119503065e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.790941343669e-09 true resid norm 3.888806692558e-10 ||Ae||/||Ax|| 5.257901377806e-13 1 KSP preconditioned resid norm 1.790941343669e-09 true resid norm 3.888806692558e-10 ||Ae||/||Ax|| 5.257901377806e-13 Otherwise: 0 SNES Function norm 6.643976542462e-01 0 KSP preconditioned resid norm 6.643976542462e-01 true resid norm 6.643976542462e-01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.725399523210e-01 true resid norm 3.166954099258e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.725399523210e-01 true resid norm 3.166954099258e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.106117940771e-13 true resid norm 2.315021615293e-12 ||Ae||/||Ax|| 7.309931065420e-13 1 KSP preconditioned resid norm 1.106117940771e-13 true resid norm 2.315021615293e-12 ||Ae||/||Ax|| 7.309931065420e-13 0 KSP preconditioned resid norm 4.929790573526e-01 true resid norm 9.951144782040e+02 ||Ae||/||Ax|| 9.951144782040e+02 1 KSP preconditioned resid norm 1.564900133045e-01 true resid norm 2.114279379086e+02 ||Ae||/||Ax|| 2.114279379086e+02 0 KSP preconditioned resid norm 1.643532578826e+03 true resid norm 3.242219706609e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.643532578826e+03 true resid norm 3.242219706609e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.414885267904e-10 true resid norm 6.046106295275e-10 ||Ae||/||Ax|| 1.864804622262e-12 1 KSP preconditioned resid norm 5.414885267904e-10 true resid norm 6.046106295275e-10 ||Ae||/||Ax|| 1.864804622262e-12 0 KSP preconditioned resid norm 2.943982335376e+03 true resid norm 5.509225819313e+06 ||Ae||/||Ax|| 5.509225819313e+06 1 KSP preconditioned resid norm 6.638944563800e-01 true resid norm 6.638944563800e-01 ||Ae||/||Ax|| 9.992426254623e-01 0 KSP preconditioned resid norm 9.123125465961e+00 true resid norm 1.382122000671e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.123125465961e+00 true resid norm 1.382122000671e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.695428248071e-12 true resid norm 3.359968818024e-12 ||Ae||/||Ax|| 2.431021875342e-12 1 KSP preconditioned resid norm 2.695428248071e-12 true resid norm 3.359968818024e-12 ||Ae||/||Ax|| 2.431021875342e-12 0 KSP preconditioned resid norm 1.640642499067e+01 true resid norm 3.058206355968e+04 ||Ae||/||Ax|| 3.058206355968e+04 0 KSP preconditioned resid norm 2.797910566205e+05 true resid norm 4.206726908841e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.797910566205e+05 true resid norm 4.206726908841e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.740791701929e-08 true resid norm 1.010734630198e-07 ||Ae||/||Ax|| 2.402662811492e-12 1 KSP preconditioned resid norm 8.740791701929e-08 true resid norm 1.010734630198e-07 ||Ae||/||Ax|| 2.402662811492e-12 0 KSP preconditioned resid norm 5.035724534228e+05 true resid norm 9.384726253260e+08 ||Ae||/||Ax|| 9.384726253260e+08 2 KSP preconditioned resid norm 6.578097894415e-01 true resid norm 6.578097894415e-01 ||Ae||/||Ax|| 9.900844550510e-01 0 KSP preconditioned resid norm 3.479843101860e+00 true resid norm 1.765526142479e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.479843101860e+00 true resid norm 1.765526142479e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.880414574516e-13 true resid norm 5.729119550389e-13 ||Ae||/||Ax|| 3.244992760257e-13 1 KSP preconditioned resid norm 6.880414574516e-13 true resid norm 5.729119550389e-13 ||Ae||/||Ax|| 3.244992760257e-13 0 KSP preconditioned resid norm 5.427680497553e+00 true resid norm 5.361948161829e+03 ||Ae||/||Ax|| 5.361948161829e+03 1 KSP preconditioned resid norm 1.300714694506e+00 true resid norm 6.605629913301e+02 ||Ae||/||Ax|| 6.605629913301e+02 0 KSP preconditioned resid norm 8.505888265686e+03 true resid norm 1.023439478635e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.505888265686e+03 true resid norm 1.023439478635e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.876815951467e-09 true resid norm 1.061881454058e-09 ||Ae||/||Ax|| 1.037561552222e-12 1 KSP preconditioned resid norm 1.876815951467e-09 true resid norm 1.061881454058e-09 ||Ae||/||Ax|| 1.037561552222e-12 0 KSP preconditioned resid norm 1.290609974182e+04 true resid norm 9.664058465504e+06 ||Ae||/||Ax|| 9.664058465504e+06 3 KSP preconditioned resid norm 6.561200836326e-01 true resid norm 6.561200836327e-01 ||Ae||/||Ax|| 9.875412404594e-01 0 KSP preconditioned resid norm 9.170586094295e+01 true resid norm 1.631125873731e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.170586094295e+01 true resid norm 1.631125873731e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.103282620964e-11 true resid norm 1.497493808500e-12 ||Ae||/||Ax|| 9.180737260177e-13 1 KSP preconditioned resid norm 1.103282620964e-11 true resid norm 1.497493808500e-12 ||Ae||/||Ax|| 9.180737260177e-13 0 KSP preconditioned resid norm 1.347370724401e+02 true resid norm 1.463309001406e+04 ||Ae||/||Ax|| 1.463309001406e+04 0 KSP preconditioned resid norm 1.777008569360e+06 true resid norm 2.049123792919e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.777008569360e+06 true resid norm 2.049123792919e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.273597544885e-07 true resid norm 3.156333476700e-08 ||Ae||/||Ax|| 1.540333232969e-12 1 KSP preconditioned resid norm 2.273597544885e-07 true resid norm 3.156333476700e-08 ||Ae||/||Ax|| 1.540333232969e-12 0 KSP preconditioned resid norm 2.610973345558e+06 true resid norm 3.067242396541e+08 ||Ae||/||Ax|| 3.067242396541e+08 4 KSP preconditioned resid norm 6.548326787335e-01 true resid norm 6.548326787335e-01 ||Ae||/||Ax|| 9.856035381047e-01 0 KSP preconditioned resid norm 6.744279341020e+01 true resid norm 1.575413893640e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.744279341020e+01 true resid norm 1.575413893640e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.596044804676e-12 true resid norm 2.188670572403e-12 ||Ae||/||Ax|| 1.389267024519e-12 1 KSP preconditioned resid norm 9.596044804676e-12 true resid norm 2.188670572403e-12 ||Ae||/||Ax|| 1.389267024519e-12 0 KSP preconditioned resid norm 9.923158845467e+01 true resid norm 2.077302460532e+04 ||Ae||/||Ax|| 2.077302460532e+04 0 KSP preconditioned resid norm 1.312863536630e+06 true resid norm 2.875558120734e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.312863536630e+06 true resid norm 2.875558120734e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.220066071362e-07 true resid norm 6.253679189569e-08 ||Ae||/||Ax|| 2.174770575659e-12 1 KSP preconditioned resid norm 2.220066071362e-07 true resid norm 6.253679189569e-08 ||Ae||/||Ax|| 2.174770575659e-12 0 KSP preconditioned resid norm 1.935695568057e+06 true resid norm 5.883388937994e+08 ||Ae||/||Ax|| 5.883388937994e+08 5 KSP preconditioned resid norm 6.544976830743e-01 true resid norm 6.544976830743e-01 ||Ae||/||Ax|| 9.850993285291e-01 0 KSP preconditioned resid norm 7.174448897397e+01 true resid norm 1.430312312462e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.174448897397e+01 true resid norm 1.430312312462e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.020678358228e-11 true resid norm 2.744384895131e-12 ||Ae||/||Ax|| 1.918731224796e-12 1 KSP preconditioned resid norm 1.020678358228e-11 true resid norm 2.744384895131e-12 ||Ae||/||Ax|| 1.918731224796e-12 0 KSP preconditioned resid norm 1.056248445749e+02 true resid norm 2.626622745496e+04 ||Ae||/||Ax|| 2.626622745496e+04 0 KSP preconditioned resid norm 1.400394797757e+06 true resid norm 3.619726511862e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.400394797757e+06 true resid norm 3.619726511862e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.193605648209e-07 true resid norm 8.494919484928e-08 ||Ae||/||Ax|| 2.346840143058e-12 1 KSP preconditioned resid norm 2.193605648209e-07 true resid norm 8.494919484928e-08 ||Ae||/||Ax|| 2.346840143058e-12 0 KSP preconditioned resid norm 2.068710085645e+06 true resid norm 7.667828660174e+08 ||Ae||/||Ax|| 7.667828660174e+08 6 KSP preconditioned resid norm 6.439039598676e-01 true resid norm 6.439039598676e-01 ||Ae||/||Ax|| 9.691544751135e-01 0 KSP preconditioned resid norm 2.179140600580e+00 true resid norm 1.989332702312e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.179140600580e+00 true resid norm 1.989332702312e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.326937782582e-13 true resid norm 4.947256796457e-13 ||Ae||/||Ax|| 2.486892610123e-13 1 KSP preconditioned resid norm 3.326937782582e-13 true resid norm 4.947256796457e-13 ||Ae||/||Ax|| 2.486892610123e-13 0 KSP preconditioned resid norm 3.444435655294e+00 true resid norm 3.156694238579e+03 ||Ae||/||Ax|| 3.156694238579e+03 1 KSP preconditioned resid norm 6.515773480483e-01 true resid norm 2.901000501958e+02 ||Ae||/||Ax|| 2.901000501958e+02 0 KSP preconditioned resid norm 3.915459779122e+03 true resid norm 5.038345879332e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.915459779122e+03 true resid norm 5.038345879332e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.039414912287e-10 true resid norm 1.678723587391e-10 ||Ae||/||Ax|| 3.331894291493e-13 1 KSP preconditioned resid norm 5.039414912287e-10 true resid norm 1.678723587391e-10 ||Ae||/||Ax|| 3.331894291493e-13 0 KSP preconditioned resid norm 5.807751000664e+03 true resid norm 1.539570200699e+06 ||Ae||/||Ax|| 1.539570200699e+06 7 KSP preconditioned resid norm 6.363395202453e-01 true resid norm 6.363395202453e-01 ||Ae||/||Ax|| 9.577690652253e-01 0 KSP preconditioned resid norm 1.052630074523e+01 true resid norm 1.781752945860e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.052630074523e+01 true resid norm 1.781752945860e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.199838493352e-12 true resid norm 7.304943596974e-13 ||Ae||/||Ax|| 4.099863347469e-13 1 KSP preconditioned resid norm 2.199838493352e-12 true resid norm 7.304943596974e-13 ||Ae||/||Ax|| 4.099863347469e-13 0 KSP preconditioned resid norm 1.564794701194e+01 true resid norm 6.589276558755e+03 ||Ae||/||Ax|| 6.589276558755e+03 0 KSP preconditioned resid norm 2.055277044182e+05 true resid norm 9.748412831200e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.055277044182e+05 true resid norm 9.748412831200e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.031500847154e-08 true resid norm 1.699803899076e-08 ||Ae||/||Ax|| 1.743672460850e-12 1 KSP preconditioned resid norm 5.031500847154e-08 true resid norm 1.699803899076e-08 ||Ae||/||Ax|| 1.743672460850e-12 0 KSP preconditioned resid norm 3.055340634218e+05 true resid norm 1.467054190881e+08 ||Ae||/||Ax|| 1.467054190881e+08 8 KSP preconditioned resid norm 6.359350697478e-01 true resid norm 6.359350697478e-01 ||Ae||/||Ax|| 9.571603175951e-01 0 KSP preconditioned resid norm 2.616392072429e+01 true resid norm 1.399164584086e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.616392072429e+01 true resid norm 1.399164584086e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.560221274332e-12 true resid norm 3.396335397864e-12 ||Ae||/||Ax|| 2.427402348868e-12 1 KSP preconditioned resid norm 8.560221274332e-12 true resid norm 3.396335397864e-12 ||Ae||/||Ax|| 2.427402348868e-12 0 KSP preconditioned resid norm 3.941467939860e+01 true resid norm 2.988458651670e+04 ||Ae||/||Ax|| 2.988458651670e+04 0 KSP preconditioned resid norm 5.488120293113e+05 true resid norm 4.111465519054e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.488120293113e+05 true resid norm 4.111465519054e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.293564612577e-07 true resid norm 9.984868100046e-08 ||Ae||/||Ax|| 2.428542341842e-12 1 KSP preconditioned resid norm 2.293564612577e-07 true resid norm 9.984868100046e-08 ||Ae||/||Ax|| 2.428542341842e-12 0 KSP preconditioned resid norm 8.476371588574e+05 true resid norm 9.130561850506e+08 ||Ae||/||Ax|| 9.130561850506e+08 9 KSP preconditioned resid norm 6.335496131622e-01 true resid norm 6.335496131621e-01 ||Ae||/||Ax|| 9.535699127067e-01 0 KSP preconditioned resid norm 1.106152616615e+02 true resid norm 1.583060778556e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.106152616615e+02 true resid norm 1.583060778556e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.805535309513e-11 true resid norm 1.673866852654e-12 ||Ae||/||Ax|| 1.057361078822e-12 1 KSP preconditioned resid norm 1.805535309513e-11 true resid norm 1.673866852654e-12 ||Ae||/||Ax|| 1.057361078822e-12 0 KSP preconditioned resid norm 1.629014513164e+02 true resid norm 1.721563662560e+04 ||Ae||/||Ax|| 1.721563662560e+04 0 KSP preconditioned resid norm 2.143400094103e+06 true resid norm 2.391592584678e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.143400094103e+06 true resid norm 2.391592584678e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.655288006358e-07 true resid norm 3.651187137683e-08 ||Ae||/||Ax|| 1.526676057233e-12 1 KSP preconditioned resid norm 3.655288006358e-07 true resid norm 3.651187137683e-08 ||Ae||/||Ax|| 1.526676057233e-12 0 KSP preconditioned resid norm 3.157145952315e+06 true resid norm 3.635185867036e+08 ||Ae||/||Ax|| 3.635185867036e+08 10 KSP preconditioned resid norm 6.286299560309e-01 true resid norm 6.286299560308e-01 ||Ae||/||Ax|| 9.461652250172e-01 0 KSP preconditioned resid norm 7.134714110556e+01 true resid norm 1.817675317602e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.134714110556e+01 true resid norm 1.817675317602e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.367511627937e-11 true resid norm 1.069313196151e-12 ||Ae||/||Ax|| 5.882861398820e-13 1 KSP preconditioned resid norm 1.367511627937e-11 true resid norm 1.069313196151e-12 ||Ae||/||Ax|| 5.882861398820e-13 0 KSP preconditioned resid norm 1.051163731361e+02 true resid norm 1.080698356605e+04 ||Ae||/||Ax|| 1.080698356605e+04 0 KSP preconditioned resid norm 1.382173007807e+06 true resid norm 1.533268566333e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.382173007807e+06 true resid norm 1.533268566333e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.409130608602e-07 true resid norm 1.986640520809e-08 ||Ae||/||Ax|| 1.295689851362e-12 1 KSP preconditioned resid norm 2.409130608602e-07 true resid norm 1.986640520809e-08 ||Ae||/||Ax|| 1.295689851362e-12 0 KSP preconditioned resid norm 2.035749163308e+06 true resid norm 2.035717866610e+08 ||Ae||/||Ax|| 2.035717866610e+08 11 KSP preconditioned resid norm 6.240432713585e-01 true resid norm 6.240432713583e-01 ||Ae||/||Ax|| 9.392617017383e-01 0 KSP preconditioned resid norm 3.844303874505e+00 true resid norm 2.025754466097e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.844303874505e+00 true resid norm 2.025754466097e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.084772487291e-13 true resid norm 2.498173588423e-13 ||Ae||/||Ax|| 1.233206506629e-13 1 KSP preconditioned resid norm 7.084772487291e-13 true resid norm 2.498173588423e-13 ||Ae||/||Ax|| 1.233206506629e-13 0 KSP preconditioned resid norm 6.693139988729e+00 true resid norm 3.219334957995e+03 ||Ae||/||Ax|| 3.219334957995e+03 1 KSP preconditioned resid norm 9.050137472631e-01 true resid norm 1.774360140771e+02 ||Ae||/||Ax|| 1.774360140771e+02 0 KSP preconditioned resid norm 9.159860571453e+02 true resid norm 2.849079799533e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.159860571453e+02 true resid norm 2.849079799533e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.715724723628e-10 true resid norm 6.479512209525e-11 ||Ae||/||Ax|| 2.274247358950e-13 1 KSP preconditioned resid norm 1.715724723628e-10 true resid norm 6.479512209525e-11 ||Ae||/||Ax|| 2.274247358950e-13 0 KSP preconditioned resid norm 1.534505089525e+03 true resid norm 7.119178925600e+05 ||Ae||/||Ax|| 7.119178925600e+05 12 KSP preconditioned resid norm 6.215082061046e-01 true resid norm 6.215082061043e-01 ||Ae||/||Ax|| 9.354461174452e-01 0 KSP preconditioned resid norm 9.582212676361e+00 true resid norm 1.600198293761e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.582212676361e+00 true resid norm 1.600198293761e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.199386304534e-12 true resid norm 2.546378223943e-12 ||Ae||/||Ax|| 1.591289175767e-12 1 KSP preconditioned resid norm 4.199386304534e-12 true resid norm 2.546378223943e-12 ||Ae||/||Ax|| 1.591289175767e-12 0 KSP preconditioned resid norm 1.605976238605e+01 true resid norm 2.420119403833e+04 ||Ae||/||Ax|| 2.420119403833e+04 0 KSP preconditioned resid norm 2.470980401312e+05 true resid norm 3.335769643727e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.470980401312e+05 true resid norm 3.335769643727e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.641301617231e-08 true resid norm 7.742484597259e-08 ||Ae||/||Ax|| 2.321048940480e-12 1 KSP preconditioned resid norm 9.641301617231e-08 true resid norm 7.742484597259e-08 ||Ae||/||Ax|| 2.321048940480e-12 0 KSP preconditioned resid norm 4.265452000486e+05 true resid norm 7.382063118236e+08 ||Ae||/||Ax|| 7.382063118236e+08 13 KSP preconditioned resid norm 6.174461224321e-01 true resid norm 6.174461224316e-01 ||Ae||/||Ax|| 9.293321830465e-01 0 KSP preconditioned resid norm 7.162749884317e+00 true resid norm 1.724169083549e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.162749884317e+00 true resid norm 1.724169083549e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.245030787092e-12 true resid norm 2.158165526832e-12 ||Ae||/||Ax|| 1.251713389032e-12 1 KSP preconditioned resid norm 3.245030787092e-12 true resid norm 2.158165526832e-12 ||Ae||/||Ax|| 1.251713389032e-12 0 KSP preconditioned resid norm 1.261531472323e+01 true resid norm 1.934063218915e+04 ||Ae||/||Ax|| 1.934063218915e+04 1 KSP preconditioned resid norm 3.138349049840e+00 true resid norm 3.196943394208e+03 ||Ae||/||Ax|| 3.196943394208e+03 0 KSP preconditioned resid norm 3.009138537539e+04 true resid norm 4.382853471310e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.009138537539e+04 true resid norm 4.382853471310e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.158503853275e-08 true resid norm 1.004172948430e-08 ||Ae||/||Ax|| 2.291139676477e-12 1 KSP preconditioned resid norm 1.158503853275e-08 true resid norm 1.004172948430e-08 ||Ae||/||Ax|| 2.291139676477e-12 0 KSP preconditioned resid norm 5.318204723101e+04 true resid norm 9.571798797533e+07 ||Ae||/||Ax|| 9.571798797533e+07 14 KSP preconditioned resid norm 6.149499196279e-01 true resid norm 6.149499196273e-01 ||Ae||/||Ax|| 9.255750915091e-01 0 KSP preconditioned resid norm 1.035088573396e+02 true resid norm 1.643475237600e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.035088573396e+02 true resid norm 1.643475237600e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.216424963006e-11 true resid norm 1.506827148268e-12 ||Ae||/||Ax|| 9.168541842274e-13 1 KSP preconditioned resid norm 1.216424963006e-11 true resid norm 1.506827148268e-12 ||Ae||/||Ax|| 9.168541842274e-13 0 KSP preconditioned resid norm 1.551868703751e+02 true resid norm 1.536835258272e+04 ||Ae||/||Ax|| 1.536835258272e+04 0 KSP preconditioned resid norm 2.005122433856e+06 true resid norm 2.139846293667e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.005122433856e+06 true resid norm 2.139846293667e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.699635663738e-07 true resid norm 3.022165390047e-08 ||Ae||/||Ax|| 1.412328258806e-12 1 KSP preconditioned resid norm 2.699635663738e-07 true resid norm 3.022165390047e-08 ||Ae||/||Ax|| 1.412328258806e-12 0 KSP preconditioned resid norm 3.006635999374e+06 true resid norm 3.017362591687e+08 ||Ae||/||Ax|| 3.017362591687e+08 15 KSP preconditioned resid norm 6.119892528016e-01 true resid norm 6.119892528010e-01 ||Ae||/||Ax|| 9.211189246225e-01 0 KSP preconditioned resid norm 7.708088998606e+01 true resid norm 1.711478746266e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.708088998606e+01 true resid norm 1.711478746266e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.060008672565e-11 true resid norm 1.607068827031e-12 ||Ae||/||Ax|| 9.389943232059e-13 1 KSP preconditioned resid norm 1.060008672565e-11 true resid norm 1.607068827031e-12 ||Ae||/||Ax|| 9.389943232059e-13 0 KSP preconditioned resid norm 1.156508047473e+02 true resid norm 1.535553658162e+04 ||Ae||/||Ax|| 1.535553658162e+04 0 KSP preconditioned resid norm 1.494713077437e+06 true resid norm 2.137562818275e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.494713077437e+06 true resid norm 2.137562818275e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.212752436065e-07 true resid norm 3.856329184915e-08 ||Ae||/||Ax|| 1.804077593390e-12 1 KSP preconditioned resid norm 2.212752436065e-07 true resid norm 3.856329184915e-08 ||Ae||/||Ax|| 1.804077593390e-12 0 KSP preconditioned resid norm 2.242894821533e+06 true resid norm 3.794045949252e+08 ||Ae||/||Ax|| 3.794045949252e+08 16 KSP preconditioned resid norm 6.117490632701e-01 true resid norm 6.117490632695e-01 ||Ae||/||Ax|| 9.207574099032e-01 0 KSP preconditioned resid norm 3.772808947649e+01 true resid norm 1.405980393316e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.772808947649e+01 true resid norm 1.405980393316e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.476258783730e-12 true resid norm 2.941069111800e-12 ||Ae||/||Ax|| 2.091827969851e-12 1 KSP preconditioned resid norm 6.476258783730e-12 true resid norm 2.941069111800e-12 ||Ae||/||Ax|| 2.091827969851e-12 0 KSP preconditioned resid norm 5.708326125111e+01 true resid norm 2.931735332166e+04 ||Ae||/||Ax|| 2.931735332166e+04 0 KSP preconditioned resid norm 7.589513697668e+05 true resid norm 4.034303627079e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.589513697668e+05 true resid norm 4.034303627079e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.910686949086e-07 true resid norm 9.440768132030e-08 ||Ae||/||Ax|| 2.340123353300e-12 1 KSP preconditioned resid norm 1.910686949086e-07 true resid norm 9.440768132030e-08 ||Ae||/||Ax|| 2.340123353300e-12 0 KSP preconditioned resid norm 1.161530456729e+06 true resid norm 8.904990807332e+08 ||Ae||/||Ax|| 8.904990807332e+08 17 KSP preconditioned resid norm 6.105492772342e-01 true resid norm 6.105492772335e-01 ||Ae||/||Ax|| 9.189515846894e-01 0 KSP preconditioned resid norm 1.151371970652e+01 true resid norm 1.840366588465e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.151371970652e+01 true resid norm 1.840366588465e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.665910011647e-12 true resid norm 3.270852369182e-13 ||Ae||/||Ax|| 1.777283063974e-13 1 KSP preconditioned resid norm 1.665910011647e-12 true resid norm 3.270852369182e-13 ||Ae||/||Ax|| 1.777283063974e-13 0 KSP preconditioned resid norm 1.982093669992e+01 true resid norm 4.092310687071e+03 ||Ae||/||Ax|| 4.092310687071e+03 1 KSP preconditioned resid norm 2.901141640394e+00 true resid norm 3.049777392969e+02 ||Ae||/||Ax|| 3.049777392969e+02 0 KSP preconditioned resid norm 5.458502977349e+03 true resid norm 4.752917147299e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.458502977349e+03 true resid norm 4.752917147299e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.762473639268e-10 true resid norm 1.958600237389e-10 ||Ae||/||Ax|| 4.120838164626e-13 1 KSP preconditioned resid norm 8.762473639268e-10 true resid norm 1.958600237389e-10 ||Ae||/||Ax|| 4.120838164626e-13 0 KSP preconditioned resid norm 8.830916923215e+03 true resid norm 1.946797262204e+06 ||Ae||/||Ax|| 1.946797262204e+06 18 KSP preconditioned resid norm 6.100978815994e-01 true resid norm 6.100978815987e-01 ||Ae||/||Ax|| 9.182721788669e-01 0 KSP preconditioned resid norm 1.293152526116e+01 true resid norm 1.770047949512e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.293152526116e+01 true resid norm 1.770047949512e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.309788070646e-12 true resid norm 1.025001129138e-12 ||Ae||/||Ax|| 5.790809957554e-13 1 KSP preconditioned resid norm 2.309788070646e-12 true resid norm 1.025001129138e-12 ||Ae||/||Ax|| 5.790809957554e-13 0 KSP preconditioned resid norm 1.985392961918e+01 true resid norm 8.566236604412e+03 ||Ae||/||Ax|| 8.566236604412e+03 1 KSP preconditioned resid norm 2.789899652666e+00 true resid norm 5.636415083587e+02 ||Ae||/||Ax|| 5.636415083587e+02 0 KSP preconditioned resid norm 1.733288768604e+04 true resid norm 7.546515122842e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.733288768604e+04 true resid norm 7.546515122842e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.025556233669e-09 true resid norm 1.580051674878e-09 ||Ae||/||Ax|| 2.093750094127e-12 1 KSP preconditioned resid norm 4.025556233669e-09 true resid norm 1.580051674878e-09 ||Ae||/||Ax|| 2.093750094127e-12 0 KSP preconditioned resid norm 2.630250151670e+04 true resid norm 1.330699007354e+07 ||Ae||/||Ax|| 1.330699007354e+07 19 KSP preconditioned resid norm 6.100476482561e-01 true resid norm 6.100476482555e-01 ||Ae||/||Ax|| 9.181965715210e-01 0 KSP preconditioned resid norm 2.339579819469e+01 true resid norm 1.408053443111e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.339579819469e+01 true resid norm 1.408053443111e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.059179601313e-12 true resid norm 3.493662559442e-12 ||Ae||/||Ax|| 2.481200253111e-12 1 KSP preconditioned resid norm 7.059179601313e-12 true resid norm 3.493662559442e-12 ||Ae||/||Ax|| 2.481200253111e-12 0 KSP preconditioned resid norm 3.611630692345e+01 true resid norm 2.951947847829e+04 ||Ae||/||Ax|| 2.951947847829e+04 0 KSP preconditioned resid norm 4.983690252370e+05 true resid norm 4.062078711009e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.983690252370e+05 true resid norm 4.062078711009e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.933795534368e-07 true resid norm 1.083017498942e-07 ||Ae||/||Ax|| 2.666165714629e-12 1 KSP preconditioned resid norm 1.933795534368e-07 true resid norm 1.083017498942e-07 ||Ae||/||Ax|| 2.666165714629e-12 0 KSP preconditioned resid norm 7.927031893526e+05 true resid norm 9.022735540634e+08 ||Ae||/||Ax|| 9.022735540634e+08 20 KSP preconditioned resid norm 6.099044961083e-01 true resid norm 6.099044961077e-01 ||Ae||/||Ax|| 9.179811099720e-01 0 KSP preconditioned resid norm 1.251927093876e+02 true resid norm 1.442674197968e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.251927093876e+02 true resid norm 1.442674197968e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.456260036321e-11 true resid norm 1.988746674151e-12 ||Ae||/||Ax|| 1.378514065721e-12 1 KSP preconditioned resid norm 1.456260036321e-11 true resid norm 1.988746674151e-12 ||Ae||/||Ax|| 1.378514065721e-12 0 KSP preconditioned resid norm 1.873393344533e+02 true resid norm 1.892014590772e+04 ||Ae||/||Ax|| 1.892014590772e+04 0 KSP preconditioned resid norm 2.424466095987e+06 true resid norm 2.622281316135e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.424466095987e+06 true resid norm 2.622281316135e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.415684007216e-07 true resid norm 3.908490203452e-08 ||Ae||/||Ax|| 1.490492335587e-12 1 KSP preconditioned resid norm 2.415684007216e-07 true resid norm 3.908490203452e-08 ||Ae||/||Ax|| 1.490492335587e-12 0 KSP preconditioned resid norm 3.626351939650e+06 true resid norm 3.890678942290e+08 ||Ae||/||Ax|| 3.890678942290e+08 21 KSP preconditioned resid norm 6.087261901598e-01 true resid norm 6.087261901594e-01 ||Ae||/||Ax|| 9.162076149261e-01 0 KSP preconditioned resid norm 4.361818580845e+01 true resid norm 1.808235581815e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.361818580845e+01 true resid norm 1.808235581815e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.784202597730e-12 true resid norm 7.762183967973e-13 ||Ae||/||Ax|| 4.292684009781e-13 1 KSP preconditioned resid norm 6.784202597730e-12 true resid norm 7.762183967973e-13 ||Ae||/||Ax|| 4.292684009781e-13 0 KSP preconditioned resid norm 6.583953430980e+01 true resid norm 7.133930076680e+03 ||Ae||/||Ax|| 7.133930076680e+03 1 KSP preconditioned resid norm 7.286222194976e+00 true resid norm 4.292142002963e+02 ||Ae||/||Ax|| 4.292142002963e+02 0 KSP preconditioned resid norm 4.338403929132e+04 true resid norm 5.595290563559e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.338403929132e+04 true resid norm 5.595290563559e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.453364725694e-09 true resid norm 6.267303520695e-10 ||Ae||/||Ax|| 1.120103317156e-12 1 KSP preconditioned resid norm 5.453364725694e-09 true resid norm 6.267303520695e-10 ||Ae||/||Ax|| 1.120103317156e-12 0 KSP preconditioned resid norm 6.512792957865e+04 true resid norm 6.434436103270e+06 ||Ae||/||Ax|| 6.434436103270e+06 22 KSP preconditioned resid norm 6.077687858338e-01 true resid norm 6.077687858336e-01 ||Ae||/||Ax|| 9.147666039296e-01 0 KSP preconditioned resid norm 3.342477813280e+01 true resid norm 1.697753978801e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.342477813280e+01 true resid norm 1.697753978801e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.150944066234e-12 true resid norm 2.293439355587e-12 ||Ae||/||Ax|| 1.350866724051e-12 1 KSP preconditioned resid norm 5.150944066234e-12 true resid norm 2.293439355587e-12 ||Ae||/||Ax|| 1.350866724051e-12 0 KSP preconditioned resid norm 5.032915530318e+01 true resid norm 2.116838708059e+04 ||Ae||/||Ax|| 2.116838708059e+04 0 KSP preconditioned resid norm 6.525917140824e+05 true resid norm 2.922908744358e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.525917140824e+05 true resid norm 2.922908744358e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.104328032200e-07 true resid norm 6.770091806491e-08 ||Ae||/||Ax|| 2.316217302219e-12 1 KSP preconditioned resid norm 1.104328032200e-07 true resid norm 6.770091806491e-08 ||Ae||/||Ax|| 2.316217302219e-12 0 KSP preconditioned resid norm 9.810237265904e+05 true resid norm 6.357251340383e+08 ||Ae||/||Ax|| 6.357251340383e+08 23 KSP preconditioned resid norm 6.069380285565e-01 true resid norm 6.069380285564e-01 ||Ae||/||Ax|| 9.135162122826e-01 0 KSP preconditioned resid norm 1.059057767841e+01 true resid norm 1.676979421356e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.059057767841e+01 true resid norm 1.676979421356e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.019380321433e-12 true resid norm 2.416866152316e-12 ||Ae||/||Ax|| 1.441202033572e-12 1 KSP preconditioned resid norm 4.019380321433e-12 true resid norm 2.416866152316e-12 ||Ae||/||Ax|| 1.441202033572e-12 0 KSP preconditioned resid norm 1.856014897240e+01 true resid norm 2.258409074155e+04 ||Ae||/||Ax|| 2.258409074155e+04 1 KSP preconditioned resid norm 4.134595169550e+00 true resid norm 3.095073305960e+03 ||Ae||/||Ax|| 3.095073305960e+03 0 KSP preconditioned resid norm 3.186685909810e+04 true resid norm 4.221911484877e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.186685909810e+04 true resid norm 4.221911484877e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.033970147546e-08 true resid norm 9.465431646552e-09 ||Ae||/||Ax|| 2.241977758287e-12 1 KSP preconditioned resid norm 1.033970147546e-08 true resid norm 9.465431646552e-09 ||Ae||/||Ax|| 2.241977758287e-12 0 KSP preconditioned resid norm 5.453066254562e+04 true resid norm 9.302387893635e+07 ||Ae||/||Ax|| 9.302387893635e+07 24 KSP preconditioned resid norm 6.067888484804e-01 true resid norm 6.067888484804e-01 ||Ae||/||Ax|| 9.132916779617e-01 0 KSP preconditioned resid norm 1.291113553227e+02 true resid norm 1.435263438672e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.291113553227e+02 true resid norm 1.435263438672e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.063613842780e-11 true resid norm 1.896260102609e-12 ||Ae||/||Ax|| 1.321193065688e-12 1 KSP preconditioned resid norm 1.063613842780e-11 true resid norm 1.896260102609e-12 ||Ae||/||Ax|| 1.321193065688e-12 0 KSP preconditioned resid norm 1.906301114234e+02 true resid norm 1.909600705660e+04 ||Ae||/||Ax|| 1.909600705660e+04 0 KSP preconditioned resid norm 2.501076355639e+06 true resid norm 2.646155443442e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.501076355639e+06 true resid norm 2.646155443442e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.359675984372e-07 true resid norm 3.948400636824e-08 ||Ae||/||Ax|| 1.492127246949e-12 1 KSP preconditioned resid norm 2.359675984372e-07 true resid norm 3.948400636824e-08 ||Ae||/||Ax|| 1.492127246949e-12 0 KSP preconditioned resid norm 3.692870740707e+06 true resid norm 3.829388910660e+08 ||Ae||/||Ax|| 3.829388910660e+08 25 KSP preconditioned resid norm 6.061458143347e-01 true resid norm 6.061458143349e-01 ||Ae||/||Ax|| 9.123238326640e-01 0 KSP preconditioned resid norm 1.898570610054e+01 true resid norm 1.765725343657e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.898570610054e+01 true resid norm 1.765725343657e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.936978050501e-12 true resid norm 6.086094877948e-13 ||Ae||/||Ax|| 3.446795901645e-13 1 KSP preconditioned resid norm 2.936978050501e-12 true resid norm 6.086094877948e-13 ||Ae||/||Ax|| 3.446795901645e-13 0 KSP preconditioned resid norm 2.890356755979e+01 true resid norm 5.632353342854e+03 ||Ae||/||Ax|| 5.632353342854e+03 1 KSP preconditioned resid norm 3.347545957016e+00 true resid norm 2.977681913510e+02 ||Ae||/||Ax|| 2.977681913510e+02 0 KSP preconditioned resid norm 1.672293494579e+04 true resid norm 3.791067067454e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.672293494579e+04 true resid norm 3.791067067454e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.688318749233e-09 true resid norm 3.706105424556e-10 ||Ae||/||Ax|| 9.775889897526e-13 1 KSP preconditioned resid norm 1.688318749233e-09 true resid norm 3.706105424556e-10 ||Ae||/||Ax|| 9.775889897526e-13 0 KSP preconditioned resid norm 2.479351467762e+04 true resid norm 3.649825941230e+06 ||Ae||/||Ax|| 3.649825941230e+06 26 KSP preconditioned resid norm 6.061203278023e-01 true resid norm 6.061203278025e-01 ||Ae||/||Ax|| 9.122854723052e-01 0 KSP preconditioned resid norm 1.113968853625e+01 true resid norm 1.391648103460e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.113968853625e+01 true resid norm 1.391648103460e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.583571291504e-12 true resid norm 3.085091090508e-12 ||Ae||/||Ax|| 2.216861491664e-12 1 KSP preconditioned resid norm 2.583571291504e-12 true resid norm 3.085091090508e-12 ||Ae||/||Ax|| 2.216861491664e-12 0 KSP preconditioned resid norm 1.817483010767e+01 true resid norm 3.036021800260e+04 ||Ae||/||Ax|| 3.036021800260e+04 0 KSP preconditioned resid norm 3.035870909300e+05 true resid norm 4.176399375582e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.035870909300e+05 true resid norm 4.176399375582e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.682916311580e-08 true resid norm 9.856631045334e-08 ||Ae||/||Ax|| 2.360078660811e-12 1 KSP preconditioned resid norm 7.682916311580e-08 true resid norm 9.856631045334e-08 ||Ae||/||Ax|| 2.360078660811e-12 0 KSP preconditioned resid norm 5.052605839103e+05 true resid norm 9.313866415726e+08 ||Ae||/||Ax|| 9.313866415726e+08 27 KSP preconditioned resid norm 6.050582839469e-01 true resid norm 6.050582839475e-01 ||Ae||/||Ax|| 9.106869659767e-01 0 KSP preconditioned resid norm 2.990447709478e+01 true resid norm 1.906982335170e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.990447709478e+01 true resid norm 1.906982335170e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.975901688762e-12 true resid norm 8.959071659844e-13 ||Ae||/||Ax|| 4.698035998874e-13 1 KSP preconditioned resid norm 3.975901688762e-12 true resid norm 8.959071659844e-13 ||Ae||/||Ax|| 4.698035998874e-13 0 KSP preconditioned resid norm 4.564797272899e+01 true resid norm 6.660826868796e+03 ||Ae||/||Ax|| 6.660826868796e+03 1 KSP preconditioned resid norm 5.741856900376e+00 true resid norm 4.316372746358e+02 ||Ae||/||Ax|| 4.316372746358e+02 0 KSP preconditioned resid norm 3.362508054185e+04 true resid norm 5.723491523243e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.362508054185e+04 true resid norm 5.723491523243e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.908925535146e-09 true resid norm 8.167920463020e-10 ||Ae||/||Ax|| 1.427087020196e-12 1 KSP preconditioned resid norm 2.908925535146e-09 true resid norm 8.167920463020e-10 ||Ae||/||Ax|| 1.427087020196e-12 0 KSP preconditioned resid norm 5.016878651847e+04 true resid norm 7.777153439460e+06 ||Ae||/||Ax|| 7.777153439460e+06 28 KSP preconditioned resid norm 6.049696067046e-01 true resid norm 6.049696067052e-01 ||Ae||/||Ax|| 9.105534958452e-01 0 KSP preconditioned resid norm 1.307515958591e+02 true resid norm 1.427906745250e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.307515958591e+02 true resid norm 1.427906745250e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.182758355330e-11 true resid norm 2.057201119195e-12 ||Ae||/||Ax|| 1.440711115091e-12 1 KSP preconditioned resid norm 1.182758355330e-11 true resid norm 2.057201119195e-12 ||Ae||/||Ax|| 1.440711115091e-12 0 KSP preconditioned resid norm 1.939032992484e+02 true resid norm 1.887943093068e+04 ||Ae||/||Ax|| 1.887943093068e+04 0 KSP preconditioned resid norm 2.532822079009e+06 true resid norm 2.617156607772e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.532822079009e+06 true resid norm 2.617156607772e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.703584245532e-07 true resid norm 3.916415744457e-08 ||Ae||/||Ax|| 1.496439201547e-12 1 KSP preconditioned resid norm 2.703584245532e-07 true resid norm 3.916415744457e-08 ||Ae||/||Ax|| 1.496439201547e-12 0 KSP preconditioned resid norm 3.755750940247e+06 true resid norm 3.657512765233e+08 ||Ae||/||Ax|| 3.657512765233e+08 29 KSP preconditioned resid norm 6.042206997356e-01 true resid norm 6.042206997367e-01 ||Ae||/||Ax|| 9.094262989567e-01 0 KSP preconditioned resid norm 1.605346211532e+01 true resid norm 1.860429376270e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.605346211532e+01 true resid norm 1.860429376270e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.246735117107e-12 true resid norm 5.084451043297e-13 ||Ae||/||Ax|| 2.732944936340e-13 1 KSP preconditioned resid norm 4.246735117107e-12 true resid norm 5.084451043297e-13 ||Ae||/||Ax|| 2.732944936340e-13 0 KSP preconditioned resid norm 2.610320208714e+01 true resid norm 4.065852706967e+03 ||Ae||/||Ax|| 4.065852706967e+03 1 KSP preconditioned resid norm 2.877850025932e+00 true resid norm 1.989585620495e+02 ||Ae||/||Ax|| 1.989585620495e+02 0 KSP preconditioned resid norm 8.393515860467e+03 true resid norm 2.716610215568e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.393515860467e+03 true resid norm 2.716610215568e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.135902017233e-09 true resid norm 1.613470301832e-10 ||Ae||/||Ax|| 5.939277900765e-13 1 KSP preconditioned resid norm 1.135902017233e-09 true resid norm 1.613470301832e-10 ||Ae||/||Ax|| 5.939277900765e-13 0 KSP preconditioned resid norm 1.249128866550e+04 true resid norm 1.357354794244e+06 ||Ae||/||Ax|| 1.357354794244e+06 30 KSP preconditioned resid norm 6.036072464106e-01 true resid norm 6.036072464106e-01 ||Ae||/||Ax|| 9.085029764221e-01 0 KSP preconditioned resid norm 2.883261330146e+00 true resid norm 3.250739645318e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.883261330146e+00 true resid norm 3.250739645318e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.235830945947e-13 true resid norm 2.317887358969e-13 ||Ae||/||Ax|| 7.130338359480e-14 1 KSP preconditioned resid norm 4.235830945947e-13 true resid norm 2.317887358969e-13 ||Ae||/||Ax|| 7.130338359480e-14 0 KSP preconditioned resid norm 4.332828306944e+00 true resid norm 4.805075908318e+02 ||Ae||/||Ax|| 4.805075908318e+02 1 KSP preconditioned resid norm 4.997777789575e-01 true resid norm 2.836772510689e+01 ||Ae||/||Ax|| 2.836772510689e+01 0 KSP preconditioned resid norm 2.719893799256e+03 true resid norm 3.835679662417e+01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.719893799256e+03 true resid norm 3.835679662417e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.870465267545e-10 true resid norm 3.805711052931e-11 ||Ae||/||Ax|| 9.921868841708e-13 1 KSP preconditioned resid norm 3.870465267545e-10 true resid norm 3.805711052931e-11 ||Ae||/||Ax|| 9.921868841708e-13 0 KSP preconditioned resid norm 4.032513880803e+03 true resid norm 4.073608352827e+05 ||Ae||/||Ax|| 4.073608352827e+05 31 KSP preconditioned resid norm 6.033915473355e-01 true resid norm 6.033915473355e-01 ||Ae||/||Ax|| 9.081783228451e-01 0 KSP preconditioned resid norm 1.291700871249e+02 true resid norm 1.391465820303e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.291700871249e+02 true resid norm 1.391465820303e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.588667833148e-11 true resid norm 1.910717598385e-12 ||Ae||/||Ax|| 1.373168906132e-12 1 KSP preconditioned resid norm 1.588667833148e-11 true resid norm 1.910717598385e-12 ||Ae||/||Ax|| 1.373168906132e-12 0 KSP preconditioned resid norm 1.910746715229e+02 true resid norm 2.020332593927e+04 ||Ae||/||Ax|| 2.020332593927e+04 0 KSP preconditioned resid norm 2.502932703828e+06 true resid norm 2.796674096377e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.502932703828e+06 true resid norm 2.796674096377e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.496637287492e-07 true resid norm 4.190289858538e-08 ||Ae||/||Ax|| 1.498311821161e-12 1 KSP preconditioned resid norm 3.496637287492e-07 true resid norm 4.190289858538e-08 ||Ae||/||Ax|| 1.498311821161e-12 0 KSP preconditioned resid norm 3.701965548826e+06 true resid norm 4.332025520420e+08 ||Ae||/||Ax|| 4.332025520420e+08 32 KSP preconditioned resid norm 6.032347872852e-01 true resid norm 6.032347872852e-01 ||Ae||/||Ax|| 9.079423797328e-01 0 KSP preconditioned resid norm 3.085017822443e+01 true resid norm 1.411525194635e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.085017822443e+01 true resid norm 1.411525194635e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.004209530886e-12 true resid norm 3.040519508209e-12 ||Ae||/||Ax|| 2.154066763927e-12 1 KSP preconditioned resid norm 6.004209530886e-12 true resid norm 3.040519508209e-12 ||Ae||/||Ax|| 2.154066763927e-12 0 KSP preconditioned resid norm 4.655296569273e+01 true resid norm 2.908584213658e+04 ||Ae||/||Ax|| 2.908584213658e+04 0 KSP preconditioned resid norm 6.283648135204e+05 true resid norm 4.002788588006e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.283648135204e+05 true resid norm 4.002788588006e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.683469568558e-07 true resid norm 9.315480748208e-08 ||Ae||/||Ax|| 2.327247753259e-12 1 KSP preconditioned resid norm 1.683469568558e-07 true resid norm 9.315480748208e-08 ||Ae||/||Ax|| 2.327247753259e-12 0 KSP preconditioned resid norm 9.613159135516e+05 true resid norm 8.862542433870e+08 ||Ae||/||Ax|| 8.862542433870e+08 33 KSP preconditioned resid norm 6.021170668776e-01 true resid norm 6.021170668776e-01 ||Ae||/||Ax|| 9.062600733603e-01 0 KSP preconditioned resid norm 2.496747114756e+01 true resid norm 1.763454064944e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.496747114756e+01 true resid norm 1.763454064944e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.750593750321e-12 true resid norm 8.749039226071e-13 ||Ae||/||Ax|| 4.961308264272e-13 1 KSP preconditioned resid norm 2.750593750321e-12 true resid norm 8.749039226071e-13 ||Ae||/||Ax|| 4.961308264272e-13 0 KSP preconditioned resid norm 3.839165429671e+01 true resid norm 8.874445848779e+03 ||Ae||/||Ax|| 8.874445848779e+03 1 KSP preconditioned resid norm 4.704365723380e+00 true resid norm 4.544637711713e+02 ||Ae||/||Ax|| 4.544637711713e+02 0 KSP preconditioned resid norm 2.117949112992e+04 true resid norm 6.068694107650e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.117949112992e+04 true resid norm 6.068694107650e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.274185237091e-09 true resid norm 1.033879257432e-09 ||Ae||/||Ax|| 1.703627236918e-12 1 KSP preconditioned resid norm 3.274185237091e-09 true resid norm 1.033879257432e-09 ||Ae||/||Ax|| 1.703627236918e-12 0 KSP preconditioned resid norm 3.155284085205e+04 true resid norm 1.026498895736e+07 ||Ae||/||Ax|| 1.026498895736e+07 34 KSP preconditioned resid norm 6.020154889627e-01 true resid norm 6.020154889627e-01 ||Ae||/||Ax|| 9.061071861335e-01 0 KSP preconditioned resid norm 6.600844705337e+01 true resid norm 1.571282551433e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.600844705337e+01 true resid norm 1.571282551433e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.127035655198e-12 true resid norm 2.123441646685e-12 ||Ae||/||Ax|| 1.351406623047e-12 1 KSP preconditioned resid norm 9.127035655198e-12 true resid norm 2.123441646685e-12 ||Ae||/||Ax|| 1.351406623047e-12 0 KSP preconditioned resid norm 9.927912839217e+01 true resid norm 1.983631149639e+04 ||Ae||/||Ax|| 1.983631149639e+04 0 KSP preconditioned resid norm 1.282366227710e+06 true resid norm 2.746699271493e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.282366227710e+06 true resid norm 2.746699271493e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.694202700277e-07 true resid norm 5.923079044524e-08 ||Ae||/||Ax|| 2.156435218809e-12 1 KSP preconditioned resid norm 1.694202700277e-07 true resid norm 5.923079044524e-08 ||Ae||/||Ax|| 2.156435218809e-12 0 KSP preconditioned resid norm 1.930685097231e+06 true resid norm 5.580347911211e+08 ||Ae||/||Ax|| 5.580347911211e+08 35 KSP preconditioned resid norm 6.019283247034e-01 true resid norm 6.019283247034e-01 ||Ae||/||Ax|| 9.059759932269e-01 0 KSP preconditioned resid norm 8.594466828936e+00 true resid norm 1.552167113542e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.594466828936e+00 true resid norm 1.552167113542e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.302259299493e-12 true resid norm 2.501851397978e-12 ||Ae||/||Ax|| 1.611844096007e-12 1 KSP preconditioned resid norm 3.302259299493e-12 true resid norm 2.501851397978e-12 ||Ae||/||Ax|| 1.611844096007e-12 0 KSP preconditioned resid norm 1.517062035319e+01 true resid norm 2.395769141617e+04 ||Ae||/||Ax|| 2.395769141617e+04 1 KSP preconditioned resid norm 3.768531609462e+00 true resid norm 3.805704144094e+03 ||Ae||/||Ax|| 3.805704144094e+03 0 KSP preconditioned resid norm 3.564523833517e+04 true resid norm 5.210768718140e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.564523833517e+04 true resid norm 5.210768718140e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.976990409103e-09 true resid norm 1.202787548197e-08 ||Ae||/||Ax|| 2.308272758316e-12 1 KSP preconditioned resid norm 9.976990409103e-09 true resid norm 1.202787548197e-08 ||Ae||/||Ax|| 2.308272758316e-12 0 KSP preconditioned resid norm 6.204405659939e+04 true resid norm 1.148661297759e+08 ||Ae||/||Ax|| 1.148661297759e+08 36 KSP preconditioned resid norm 6.018961864301e-01 true resid norm 6.018961864301e-01 ||Ae||/||Ax|| 9.059276211819e-01 0 KSP preconditioned resid norm 1.168188962338e+02 true resid norm 1.462216988869e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.168188962338e+02 true resid norm 1.462216988869e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.418645682499e-11 true resid norm 2.041023520946e-12 ||Ae||/||Ax|| 1.395841750221e-12 1 KSP preconditioned resid norm 1.418645682499e-11 true resid norm 2.041023520946e-12 ||Ae||/||Ax|| 1.395841750221e-12 0 KSP preconditioned resid norm 1.754388599855e+02 true resid norm 1.921631796353e+04 ||Ae||/||Ax|| 1.921631796353e+04 0 KSP preconditioned resid norm 2.263469795458e+06 true resid norm 2.662858824807e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.263469795458e+06 true resid norm 2.662858824807e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.716948113752e-07 true resid norm 4.419656110223e-08 ||Ae||/||Ax|| 1.659741053130e-12 1 KSP preconditioned resid norm 2.716948113752e-07 true resid norm 4.419656110223e-08 ||Ae||/||Ax|| 1.659741053130e-12 0 KSP preconditioned resid norm 3.398850621172e+06 true resid norm 4.296111918873e+08 ||Ae||/||Ax|| 4.296111918873e+08 37 KSP preconditioned resid norm 6.018568422787e-01 true resid norm 6.018568422787e-01 ||Ae||/||Ax|| 9.058684034059e-01 0 KSP preconditioned resid norm 1.034549019132e+01 true resid norm 1.592038204875e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.034549019132e+01 true resid norm 1.592038204875e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.000967723181e-12 true resid norm 2.596964945857e-12 ||Ae||/||Ax|| 1.631220242017e-12 1 KSP preconditioned resid norm 6.000967723181e-12 true resid norm 2.596964945857e-12 ||Ae||/||Ax|| 1.631220242017e-12 0 KSP preconditioned resid norm 1.829726167467e+01 true resid norm 2.244731226201e+04 ||Ae||/||Ax|| 2.244731226201e+04 1 KSP preconditioned resid norm 3.636192348103e+00 true resid norm 2.423089425503e+03 ||Ae||/||Ax|| 2.423089425503e+03 0 KSP preconditioned resid norm 2.212926062660e+04 true resid norm 3.283730406261e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.212926062660e+04 true resid norm 3.283730406261e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.403141259045e-09 true resid norm 8.329503216749e-09 ||Ae||/||Ax|| 2.536597767243e-12 1 KSP preconditioned resid norm 9.403141259045e-09 true resid norm 8.329503216749e-09 ||Ae||/||Ax|| 2.536597767243e-12 0 KSP preconditioned resid norm 3.981873087265e+04 true resid norm 7.213540526219e+07 ||Ae||/||Ax|| 7.213540526219e+07 38 KSP preconditioned resid norm 6.018155290858e-01 true resid norm 6.018155290858e-01 ||Ae||/||Ax|| 9.058062219810e-01 0 KSP preconditioned resid norm 1.349486872589e+01 true resid norm 1.610396989430e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.349486872589e+01 true resid norm 1.610396989430e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.175356860142e-12 true resid norm 2.581123862449e-12 ||Ae||/||Ax|| 1.602787312315e-12 1 KSP preconditioned resid norm 6.175356860142e-12 true resid norm 2.581123862449e-12 ||Ae||/||Ax|| 1.602787312315e-12 0 KSP preconditioned resid norm 2.277553440852e+01 true resid norm 2.133686523868e+04 ||Ae||/||Ax|| 2.133686523868e+04 1 KSP preconditioned resid norm 4.289217345014e+00 true resid norm 2.171775997019e+03 ||Ae||/||Ax|| 2.171775997019e+03 0 KSP preconditioned resid norm 2.724991496114e+04 true resid norm 2.927865309940e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.724991496114e+04 true resid norm 2.927865309940e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.241609895331e-08 true resid norm 7.411776807546e-09 ||Ae||/||Ax|| 2.531460987083e-12 1 KSP preconditioned resid norm 1.241609895331e-08 true resid norm 7.411776807546e-09 ||Ae||/||Ax|| 2.531460987083e-12 0 KSP preconditioned resid norm 4.577058668061e+04 true resid norm 6.399361173665e+07 ||Ae||/||Ax|| 6.399361173665e+07 39 KSP preconditioned resid norm 6.018052851232e-01 true resid norm 6.018052851232e-01 ||Ae||/||Ax|| 9.057908035601e-01 0 KSP preconditioned resid norm 9.545633930615e+00 true resid norm 1.399666424034e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.545633930615e+00 true resid norm 1.399666424034e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.225392454749e-12 true resid norm 3.086870800377e-12 ||Ae||/||Ax|| 2.205433199919e-12 1 KSP preconditioned resid norm 3.225392454749e-12 true resid norm 3.086870800377e-12 ||Ae||/||Ax|| 2.205433199919e-12 0 KSP preconditioned resid norm 1.652136392969e+01 true resid norm 3.005162698204e+04 ||Ae||/||Ax|| 3.005162698204e+04 0 KSP preconditioned resid norm 2.808361970186e+05 true resid norm 4.134441311453e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.808361970186e+05 true resid norm 4.134441311453e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.960616496849e-08 true resid norm 9.365904867226e-08 ||Ae||/||Ax|| 2.265337481338e-12 1 KSP preconditioned resid norm 8.960616496849e-08 true resid norm 9.365904867226e-08 ||Ae||/||Ax|| 2.265337481338e-12 0 KSP preconditioned resid norm 4.897440548135e+05 true resid norm 9.218837674986e+08 ||Ae||/||Ax|| 9.218837674986e+08 40 KSP preconditioned resid norm 6.015950872280e-01 true resid norm 6.015950872280e-01 ||Ae||/||Ax|| 9.054744299338e-01 0 KSP preconditioned resid norm 1.144394898101e+01 true resid norm 1.814391014800e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.144394898101e+01 true resid norm 1.814391014800e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.520795319563e-12 true resid norm 8.121754756768e-13 ||Ae||/||Ax|| 4.476297937169e-13 1 KSP preconditioned resid norm 2.520795319563e-12 true resid norm 8.121754756768e-13 ||Ae||/||Ax|| 4.476297937169e-13 0 KSP preconditioned resid norm 1.858758832997e+01 true resid norm 7.522048855661e+03 ||Ae||/||Ax|| 7.522048855661e+03 1 KSP preconditioned resid norm 2.518949764582e+00 true resid norm 4.604287589273e+02 ||Ae||/||Ax|| 4.604287589273e+02 0 KSP preconditioned resid norm 1.151661799541e+04 true resid norm 6.314749970776e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.151661799541e+04 true resid norm 6.314749970776e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.557958051927e-09 true resid norm 1.135574112491e-09 ||Ae||/||Ax|| 1.798288321384e-12 1 KSP preconditioned resid norm 2.557958051927e-09 true resid norm 1.135574112491e-09 ||Ae||/||Ax|| 1.798288321384e-12 0 KSP preconditioned resid norm 1.753433812926e+04 true resid norm 1.065913803144e+07 ||Ae||/||Ax|| 1.065913803144e+07 41 KSP preconditioned resid norm 6.015090207046e-01 true resid norm 6.015090207046e-01 ||Ae||/||Ax|| 9.053448892547e-01 0 KSP preconditioned resid norm 1.258660956657e+02 true resid norm 1.438750764647e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.258660956657e+02 true resid norm 1.438750764647e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.197701486962e-11 true resid norm 1.773092241127e-12 ||Ae||/||Ax|| 1.232383179002e-12 1 KSP preconditioned resid norm 1.197701486962e-11 true resid norm 1.773092241127e-12 ||Ae||/||Ax|| 1.232383179002e-12 0 KSP preconditioned resid norm 1.867078416778e+02 true resid norm 1.892221435105e+04 ||Ae||/||Ax|| 1.892221435105e+04 0 KSP preconditioned resid norm 2.438197842595e+06 true resid norm 2.622741836494e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.438197842595e+06 true resid norm 2.622741836494e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.337308579470e-07 true resid norm 3.676792841539e-08 ||Ae||/||Ax|| 1.401888966111e-12 1 KSP preconditioned resid norm 2.337308579470e-07 true resid norm 3.676792841539e-08 ||Ae||/||Ax|| 1.401888966111e-12 0 KSP preconditioned resid norm 3.615812631507e+06 true resid norm 3.864011886426e+08 ||Ae||/||Ax|| 3.864011886426e+08 42 KSP preconditioned resid norm 6.010305638787e-01 true resid norm 6.010305638787e-01 ||Ae||/||Ax|| 9.046247530187e-01 0 KSP preconditioned resid norm 1.601892860278e+01 true resid norm 1.620093141150e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.601892860278e+01 true resid norm 1.620093141150e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.906376270553e-12 true resid norm 2.341203654775e-12 ||Ae||/||Ax|| 1.445104355613e-12 1 KSP preconditioned resid norm 3.906376270553e-12 true resid norm 2.341203654775e-12 ||Ae||/||Ax|| 1.445104355613e-12 0 KSP preconditioned resid norm 2.477212981787e+01 true resid norm 2.156064096911e+04 ||Ae||/||Ax|| 2.156064096911e+04 1 KSP preconditioned resid norm 4.331360143003e+00 true resid norm 1.728804931154e+03 ||Ae||/||Ax|| 1.728804931154e+03 0 KSP preconditioned resid norm 2.698079436494e+04 true resid norm 2.294109252400e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.698079436494e+04 true resid norm 2.294109252400e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.349797339796e-09 true resid norm 5.432498245662e-09 ||Ae||/||Ax|| 2.368020720887e-12 1 KSP preconditioned resid norm 8.349797339796e-09 true resid norm 5.432498245662e-09 ||Ae||/||Ax|| 2.368020720887e-12 0 KSP preconditioned resid norm 4.215514246356e+04 true resid norm 4.982999912105e+07 ||Ae||/||Ax|| 4.982999912105e+07 43 KSP preconditioned resid norm 6.006234170649e-01 true resid norm 6.006234170649e-01 ||Ae||/||Ax|| 9.040119471017e-01 0 KSP preconditioned resid norm 4.463958619807e+01 true resid norm 1.574622602196e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.463958619807e+01 true resid norm 1.574622602196e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.878231775653e-12 true resid norm 2.441357046325e-12 ||Ae||/||Ax|| 1.550439478591e-12 1 KSP preconditioned resid norm 5.878231775653e-12 true resid norm 2.441357046325e-12 ||Ae||/||Ax|| 1.550439478591e-12 0 KSP preconditioned resid norm 6.644535573814e+01 true resid norm 2.258145514606e+04 ||Ae||/||Ax|| 2.258145514606e+04 0 KSP preconditioned resid norm 8.735448534613e+05 true resid norm 3.116818060412e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.735448534613e+05 true resid norm 3.116818060412e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.396527101691e-07 true resid norm 7.278451382343e-08 ||Ae||/||Ax|| 2.335218559848e-12 1 KSP preconditioned resid norm 1.396527101691e-07 true resid norm 7.278451382343e-08 ||Ae||/||Ax|| 2.335218559848e-12 0 KSP preconditioned resid norm 1.306405480468e+06 true resid norm 6.715521545148e+08 ||Ae||/||Ax|| 6.715521545148e+08 44 KSP preconditioned resid norm 5.989943047846e-01 true resid norm 5.989943047846e-01 ||Ae||/||Ax|| 9.015599332062e-01 0 KSP preconditioned resid norm 7.801033326842e+00 true resid norm 1.858930249500e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.801033326842e+00 true resid norm 1.858930249500e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.080604064893e-12 true resid norm 1.090544444514e-12 ||Ae||/||Ax|| 5.866516211717e-13 1 KSP preconditioned resid norm 3.080604064893e-12 true resid norm 1.090544444514e-12 ||Ae||/||Ax|| 5.866516211717e-13 0 KSP preconditioned resid norm 1.379151864335e+01 true resid norm 9.550188217064e+03 ||Ae||/||Ax|| 9.550188217064e+03 1 KSP preconditioned resid norm 2.496696302386e+00 true resid norm 9.139746581256e+02 ||Ae||/||Ax|| 9.139746581256e+02 0 KSP preconditioned resid norm 9.828395655932e+03 true resid norm 1.265631564011e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.828395655932e+03 true resid norm 1.265631564011e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.186882436309e-09 true resid norm 2.816213825702e-09 ||Ae||/||Ax|| 2.225145062578e-12 1 KSP preconditioned resid norm 5.186882436309e-09 true resid norm 2.816213825702e-09 ||Ae||/||Ax|| 2.225145062578e-12 0 KSP preconditioned resid norm 1.683202118005e+04 true resid norm 2.519632014187e+07 ||Ae||/||Ax|| 2.519632014187e+07 45 KSP preconditioned resid norm 5.988241351926e-01 true resid norm 5.988241351925e-01 ||Ae||/||Ax|| 9.013038070881e-01 0 KSP preconditioned resid norm 9.447428573545e+00 true resid norm 1.429955730938e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.447428573545e+00 true resid norm 1.429955730938e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.374949556633e-12 true resid norm 3.254725539134e-12 ||Ae||/||Ax|| 2.276102307725e-12 1 KSP preconditioned resid norm 4.374949556633e-12 true resid norm 3.254725539134e-12 ||Ae||/||Ax|| 2.276102307725e-12 0 KSP preconditioned resid norm 1.683315376191e+01 true resid norm 2.939854160002e+04 ||Ae||/||Ax|| 2.939854160002e+04 0 KSP preconditioned resid norm 2.744710812966e+05 true resid norm 4.045045515226e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.744710812966e+05 true resid norm 4.045045515226e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.042493610744e-07 true resid norm 9.586945287664e-08 ||Ae||/||Ax|| 2.370046332378e-12 1 KSP preconditioned resid norm 1.042493610744e-07 true resid norm 9.586945287664e-08 ||Ae||/||Ax|| 2.370046332378e-12 0 KSP preconditioned resid norm 4.926385836574e+05 true resid norm 9.016080040954e+08 ||Ae||/||Ax|| 9.016080040954e+08 46 KSP preconditioned resid norm 5.976964499873e-01 true resid norm 5.976964499873e-01 ||Ae||/||Ax|| 8.996065024723e-01 0 KSP preconditioned resid norm 5.383367053747e+01 true resid norm 1.764260162117e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.383367053747e+01 true resid norm 1.764260162117e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.982003528687e-12 true resid norm 9.988259842004e-13 ||Ae||/||Ax|| 5.661443848520e-13 1 KSP preconditioned resid norm 7.982003528687e-12 true resid norm 9.988259842004e-13 ||Ae||/||Ax|| 5.661443848520e-13 0 KSP preconditioned resid norm 7.967851250204e+01 true resid norm 8.835407003521e+03 ||Ae||/||Ax|| 8.835407003521e+03 0 KSP preconditioned resid norm 1.040087569810e+06 true resid norm 1.273374485100e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.040087569810e+06 true resid norm 1.273374485100e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.866589553920e-07 true resid norm 1.610309314332e-08 ||Ae||/||Ax|| 1.264599953253e-12 1 KSP preconditioned resid norm 1.866589553920e-07 true resid norm 1.610309314332e-08 ||Ae||/||Ax|| 1.264599953253e-12 0 KSP preconditioned resid norm 1.535186941089e+06 true resid norm 1.559914877523e+08 ||Ae||/||Ax|| 1.559914877523e+08 47 KSP preconditioned resid norm 5.974396534595e-01 true resid norm 5.974396534595e-01 ||Ae||/||Ax|| 8.992199921858e-01 0 KSP preconditioned resid norm 1.228150074723e+02 true resid norm 1.476952018797e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.228150074723e+02 true resid norm 1.476952018797e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.984957681616e-11 true resid norm 1.797935076021e-12 ||Ae||/||Ax|| 1.217328019556e-12 1 KSP preconditioned resid norm 1.984957681616e-11 true resid norm 1.797935076021e-12 ||Ae||/||Ax|| 1.217328019556e-12 0 KSP preconditioned resid norm 1.811463975130e+02 true resid norm 1.780914291385e+04 ||Ae||/||Ax|| 1.780914291385e+04 0 KSP preconditioned resid norm 2.379005322375e+06 true resid norm 2.472348820548e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.379005322375e+06 true resid norm 2.472348820548e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.010884330048e-07 true resid norm 3.569588576897e-08 ||Ae||/||Ax|| 1.443804590691e-12 1 KSP preconditioned resid norm 4.010884330048e-07 true resid norm 3.569588576897e-08 ||Ae||/||Ax|| 1.443804590691e-12 0 KSP preconditioned resid norm 3.509863401975e+06 true resid norm 3.439767658288e+08 ||Ae||/||Ax|| 3.439767658288e+08 48 KSP preconditioned resid norm 5.967070550787e-01 true resid norm 5.967070550787e-01 ||Ae||/||Ax|| 8.981173417231e-01 0 KSP preconditioned resid norm 1.012539150796e+01 true resid norm 1.765630637766e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.012539150796e+01 true resid norm 1.765630637766e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.017922063749e-12 true resid norm 1.697008580802e-12 ||Ae||/||Ax|| 9.611345343154e-13 1 KSP preconditioned resid norm 3.017922063749e-12 true resid norm 1.697008580802e-12 ||Ae||/||Ax|| 9.611345343154e-13 0 KSP preconditioned resid norm 1.749232306387e+01 true resid norm 1.597340655478e+04 ||Ae||/||Ax|| 1.597340655478e+04 1 KSP preconditioned resid norm 3.684080674752e+00 true resid norm 1.813668746237e+03 ||Ae||/||Ax|| 1.813668746237e+03 0 KSP preconditioned resid norm 2.013743165402e+04 true resid norm 2.471319024569e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.013743165402e+04 true resid norm 2.471319024569e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.952352169099e-09 true resid norm 5.601376542559e-09 ||Ae||/||Ax|| 2.266553401998e-12 1 KSP preconditioned resid norm 4.952352169099e-09 true resid norm 5.601376542559e-09 ||Ae||/||Ax|| 2.266553401998e-12 0 KSP preconditioned resid norm 3.294847400007e+04 true resid norm 5.308988230187e+07 ||Ae||/||Ax|| 5.308988230187e+07 49 KSP preconditioned resid norm 5.964515532433e-01 true resid norm 5.964515532433e-01 ||Ae||/||Ax|| 8.977327801074e-01 0 KSP preconditioned resid norm 1.093052977323e+01 true resid norm 1.523465150496e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.093052977323e+01 true resid norm 1.523465150496e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.866847176528e-12 true resid norm 2.802894812225e-12 ||Ae||/||Ax|| 1.839815509605e-12 1 KSP preconditioned resid norm 2.866847176528e-12 true resid norm 2.802894812225e-12 ||Ae||/||Ax|| 1.839815509605e-12 0 KSP preconditioned resid norm 1.813239863676e+01 true resid norm 2.649037958872e+04 ||Ae||/||Ax|| 2.649037958872e+04 1 KSP preconditioned resid norm 4.704200672868e+00 true resid norm 4.271453532090e+03 ||Ae||/||Ax|| 4.271453532090e+03 0 KSP preconditioned resid norm 4.372886205198e+04 true resid norm 5.835849696038e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.372886205198e+04 true resid norm 5.835849696038e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.133182203665e-09 true resid norm 1.333609063719e-08 ||Ae||/||Ax|| 2.285201184369e-12 1 KSP preconditioned resid norm 9.133182203665e-09 true resid norm 1.333609063719e-08 ||Ae||/||Ax|| 2.285201184369e-12 0 KSP preconditioned resid norm 7.260583406741e+04 true resid norm 1.294173995053e+08 ||Ae||/||Ax|| 1.294173995053e+08 50 KSP preconditioned resid norm 5.960049857385e-01 true resid norm 5.960049857384e-01 ||Ae||/||Ax|| 8.970606412129e-01 0 KSP preconditioned resid norm 1.411765679476e+01 true resid norm 1.767166683640e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.411765679476e+01 true resid norm 1.767166683640e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.790271420127e-12 true resid norm 1.520411942802e-12 ||Ae||/||Ax|| 8.603670252939e-13 1 KSP preconditioned resid norm 3.790271420127e-12 true resid norm 1.520411942802e-12 ||Ae||/||Ax|| 8.603670252939e-13 0 KSP preconditioned resid norm 2.321069451170e+01 true resid norm 1.425848084877e+04 ||Ae||/||Ax|| 1.425848084877e+04 1 KSP preconditioned resid norm 3.191110393758e+00 true resid norm 8.938658647825e+02 ||Ae||/||Ax|| 8.938658647825e+02 0 KSP preconditioned resid norm 1.423075913273e+04 true resid norm 1.180447594766e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.423075913273e+04 true resid norm 1.180447594766e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.329162150964e-09 true resid norm 2.697398221869e-09 ||Ae||/||Ax|| 2.285063931537e-12 1 KSP preconditioned resid norm 4.329162150964e-09 true resid norm 2.697398221869e-09 ||Ae||/||Ax|| 2.285063931537e-12 0 KSP preconditioned resid norm 2.254311405745e+04 true resid norm 2.474682478603e+07 ||Ae||/||Ax|| 2.474682478603e+07 51 KSP preconditioned resid norm 5.958843670909e-01 true resid norm 5.958843670909e-01 ||Ae||/||Ax|| 8.968790953468e-01 0 KSP preconditioned resid norm 1.092502887893e+01 true resid norm 1.489476681404e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.092502887893e+01 true resid norm 1.489476681404e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.018879028103e-12 true resid norm 2.959819346932e-12 ||Ae||/||Ax|| 1.987153866782e-12 1 KSP preconditioned resid norm 5.018879028103e-12 true resid norm 2.959819346932e-12 ||Ae||/||Ax|| 1.987153866782e-12 0 KSP preconditioned resid norm 1.888449217337e+01 true resid norm 2.739572864219e+04 ||Ae||/||Ax|| 2.739572864219e+04 1 KSP preconditioned resid norm 4.035613628494e+00 true resid norm 3.621007811356e+03 ||Ae||/||Ax|| 3.621007811356e+03 0 KSP preconditioned resid norm 3.565338989617e+04 true resid norm 4.925936392346e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.565338989617e+04 true resid norm 4.925936392346e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.046410122204e-08 true resid norm 1.126961437319e-08 ||Ae||/||Ax|| 2.287811590645e-12 1 KSP preconditioned resid norm 1.046410122204e-08 true resid norm 1.126961437319e-08 ||Ae||/||Ax|| 2.287811590645e-12 0 KSP preconditioned resid norm 6.286318510382e+04 true resid norm 1.092621103884e+08 ||Ae||/||Ax|| 1.092621103884e+08 52 KSP preconditioned resid norm 5.957958851724e-01 true resid norm 5.957958851724e-01 ||Ae||/||Ax|| 8.967459192015e-01 0 KSP preconditioned resid norm 1.211500138028e+02 true resid norm 1.470634192348e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.211500138028e+02 true resid norm 1.470634192348e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.975615407437e-11 true resid norm 1.742509977986e-12 ||Ae||/||Ax|| 1.184869756906e-12 1 KSP preconditioned resid norm 1.975615407437e-11 true resid norm 1.742509977986e-12 ||Ae||/||Ax|| 1.184869756906e-12 0 KSP preconditioned resid norm 1.783437433640e+02 true resid norm 1.770927873907e+04 ||Ae||/||Ax|| 1.770927873907e+04 0 KSP preconditioned resid norm 2.346650900918e+06 true resid norm 2.459469721678e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.346650900918e+06 true resid norm 2.459469721678e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.012877817092e-07 true resid norm 3.364140079131e-08 ||Ae||/||Ax|| 1.367831467685e-12 1 KSP preconditioned resid norm 4.012877817092e-07 true resid norm 3.364140079131e-08 ||Ae||/||Ax|| 1.367831467685e-12 0 KSP preconditioned resid norm 3.453647937892e+06 true resid norm 3.451901379566e+08 ||Ae||/||Ax|| 3.451901379566e+08 53 KSP preconditioned resid norm 5.954303411293e-01 true resid norm 5.954303411293e-01 ||Ae||/||Ax|| 8.961957305597e-01 0 KSP preconditioned resid norm 5.715319337158e+01 true resid norm 1.713539825575e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.715319337158e+01 true resid norm 1.713539825575e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.897594180720e-12 true resid norm 9.392050869917e-13 ||Ae||/||Ax|| 5.481081168781e-13 1 KSP preconditioned resid norm 8.897594180720e-12 true resid norm 9.392050869917e-13 ||Ae||/||Ax|| 5.481081168781e-13 0 KSP preconditioned resid norm 8.421037088804e+01 true resid norm 9.203195248415e+03 ||Ae||/||Ax|| 9.203195248415e+03 0 KSP preconditioned resid norm 1.103266320611e+06 true resid norm 1.326262551612e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.103266320611e+06 true resid norm 1.326262551612e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.842280857507e-07 true resid norm 1.636640643909e-08 ||Ae||/||Ax|| 1.234024621987e-12 1 KSP preconditioned resid norm 1.842280857507e-07 true resid norm 1.636640643909e-08 ||Ae||/||Ax|| 1.234024621987e-12 0 KSP preconditioned resid norm 1.623605198021e+06 true resid norm 1.607960339473e+08 ||Ae||/||Ax|| 1.607960339473e+08 54 KSP preconditioned resid norm 5.949923440921e-01 true resid norm 5.949923440921e-01 ||Ae||/||Ax|| 8.955364912706e-01 0 KSP preconditioned resid norm 1.061464977350e+01 true resid norm 1.835872043325e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.061464977350e+01 true resid norm 1.835872043325e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.793279972453e-12 true resid norm 3.305229259383e-13 ||Ae||/||Ax|| 1.800359274166e-13 1 KSP preconditioned resid norm 1.793279972453e-12 true resid norm 3.305229259383e-13 ||Ae||/||Ax|| 1.800359274166e-13 0 KSP preconditioned resid norm 1.834029670367e+01 true resid norm 3.861587136413e+03 ||Ae||/||Ax|| 3.861587136413e+03 1 KSP preconditioned resid norm 2.680650618945e+00 true resid norm 2.982419314611e+02 ||Ae||/||Ax|| 2.982419314611e+02 0 KSP preconditioned resid norm 6.846891275509e+03 true resid norm 4.640227862039e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.846891275509e+03 true resid norm 4.640227862039e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.024701703837e-09 true resid norm 1.644112819257e-10 ||Ae||/||Ax|| 3.543172594405e-13 1 KSP preconditioned resid norm 1.024701703837e-09 true resid norm 1.644112819257e-10 ||Ae||/||Ax|| 3.543172594405e-13 0 KSP preconditioned resid norm 1.072754643541e+04 true resid norm 1.529713354040e+06 ||Ae||/||Ax|| 1.529713354040e+06 55 KSP preconditioned resid norm 5.947851576153e-01 true resid norm 5.947851576152e-01 ||Ae||/||Ax|| 8.952246501984e-01 0 KSP preconditioned resid norm 1.125730446931e+02 true resid norm 1.556173812245e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.125730446931e+02 true resid norm 1.556173812245e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.469897153038e-11 true resid norm 1.731757462454e-12 ||Ae||/||Ax|| 1.112830359197e-12 1 KSP preconditioned resid norm 1.469897153038e-11 true resid norm 1.731757462454e-12 ||Ae||/||Ax|| 1.112830359197e-12 0 KSP preconditioned resid norm 1.668273278200e+02 true resid norm 1.632173942149e+04 ||Ae||/||Ax|| 1.632173942149e+04 0 KSP preconditioned resid norm 2.179151939112e+06 true resid norm 2.267782824374e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.179151939112e+06 true resid norm 2.267782824374e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.847526869939e-07 true resid norm 3.227848451342e-08 ||Ae||/||Ax|| 1.423349897816e-12 1 KSP preconditioned resid norm 2.847526869939e-07 true resid norm 3.227848451342e-08 ||Ae||/||Ax|| 1.423349897816e-12 0 KSP preconditioned resid norm 3.226961147785e+06 true resid norm 3.144514013843e+08 ||Ae||/||Ax|| 3.144514013843e+08 56 KSP preconditioned resid norm 5.943009473236e-01 true resid norm 5.943009473235e-01 ||Ae||/||Ax|| 8.944958542904e-01 0 KSP preconditioned resid norm 7.363443081780e+01 true resid norm 1.757542262235e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.363443081780e+01 true resid norm 1.757542262235e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.178069690966e-11 true resid norm 1.190151633396e-12 ||Ae||/||Ax|| 6.771681449541e-13 1 KSP preconditioned resid norm 1.178069690966e-11 true resid norm 1.190151633396e-12 ||Ae||/||Ax|| 6.771681449541e-13 0 KSP preconditioned resid norm 1.090924087373e+02 true resid norm 1.091377024910e+04 ||Ae||/||Ax|| 1.091377024910e+04 0 KSP preconditioned resid norm 1.419669181163e+06 true resid norm 1.534117214607e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.419669181163e+06 true resid norm 1.534117214607e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.793307000436e-07 true resid norm 2.115715522194e-08 ||Ae||/||Ax|| 1.379109433132e-12 1 KSP preconditioned resid norm 1.793307000436e-07 true resid norm 2.115715522194e-08 ||Ae||/||Ax|| 1.379109433132e-12 0 KSP preconditioned resid norm 2.101655795427e+06 true resid norm 2.051504710492e+08 ||Ae||/||Ax|| 2.051504710492e+08 57 KSP preconditioned resid norm 5.937338810967e-01 true resid norm 5.937338810966e-01 ||Ae||/||Ax|| 8.936423500324e-01 0 KSP preconditioned resid norm 5.540540796433e+00 true resid norm 1.771708196255e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.540540796433e+00 true resid norm 1.771708196255e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.900895699938e-13 true resid norm 3.966205335163e-13 ||Ae||/||Ax|| 2.238633508355e-13 1 KSP preconditioned resid norm 9.900895699938e-13 true resid norm 3.966205335163e-13 ||Ae||/||Ax|| 2.238633508355e-13 0 KSP preconditioned resid norm 9.701783015791e+00 true resid norm 4.626355063667e+03 ||Ae||/||Ax|| 4.626355063667e+03 1 KSP preconditioned resid norm 1.731539506149e+00 true resid norm 4.020164242588e+02 ||Ae||/||Ax|| 4.020164242588e+02 0 KSP preconditioned resid norm 4.360695698079e+03 true resid norm 6.007582914621e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.360695698079e+03 true resid norm 6.007582914621e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.740768116089e-10 true resid norm 2.018820313798e-10 ||Ae||/||Ax|| 3.360453517644e-13 1 KSP preconditioned resid norm 6.740768116089e-10 true resid norm 2.018820313798e-10 ||Ae||/||Ax|| 3.360453517644e-13 0 KSP preconditioned resid norm 6.754387403272e+03 true resid norm 2.034041489697e+06 ||Ae||/||Ax|| 2.034041489697e+06 58 KSP preconditioned resid norm 5.932312048038e-01 true resid norm 5.932312048037e-01 ||Ae||/||Ax|| 8.928857605266e-01 0 KSP preconditioned resid norm 2.194320843042e+01 true resid norm 1.869287614685e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.194320843042e+01 true resid norm 1.869287614685e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.296101831586e-12 true resid norm 5.421633987559e-13 ||Ae||/||Ax|| 2.900374423373e-13 1 KSP preconditioned resid norm 5.296101831586e-12 true resid norm 5.421633987559e-13 ||Ae||/||Ax|| 2.900374423373e-13 0 KSP preconditioned resid norm 3.423876919685e+01 true resid norm 4.547328997714e+03 ||Ae||/||Ax|| 4.547328997714e+03 1 KSP preconditioned resid norm 3.995343397614e+00 true resid norm 2.761588321024e+02 ||Ae||/||Ax|| 2.761588321024e+02 0 KSP preconditioned resid norm 2.142308971213e+04 true resid norm 3.951713072929e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.142308971213e+04 true resid norm 3.951713072929e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.296595003158e-09 true resid norm 3.330987235677e-10 ||Ae||/||Ax|| 8.429223413248e-13 1 KSP preconditioned resid norm 3.296595003158e-09 true resid norm 3.330987235677e-10 ||Ae||/||Ax|| 8.429223413248e-13 0 KSP preconditioned resid norm 3.202913929430e+04 true resid norm 3.173564174806e+06 ||Ae||/||Ax|| 3.173564174806e+06 59 KSP preconditioned resid norm 5.932111446241e-01 true resid norm 5.932111446240e-01 ||Ae||/||Ax|| 8.928555674944e-01 0 KSP preconditioned resid norm 1.324767095623e+02 true resid norm 1.405154609000e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.324767095623e+02 true resid norm 1.405154609000e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.460794873924e-11 true resid norm 1.931722813340e-12 ||Ae||/||Ax|| 1.374740402919e-12 1 KSP preconditioned resid norm 1.460794873924e-11 true resid norm 1.931722813340e-12 ||Ae||/||Ax|| 1.374740402919e-12 0 KSP preconditioned resid norm 1.967181449773e+02 true resid norm 1.911658248966e+04 ||Ae||/||Ax|| 1.911658248966e+04 0 KSP preconditioned resid norm 2.566330314698e+06 true resid norm 2.649537984418e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.566330314698e+06 true resid norm 2.649537984418e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.012663459242e-07 true resid norm 3.698257725532e-08 ||Ae||/||Ax|| 1.395812306629e-12 1 KSP preconditioned resid norm 3.012663459242e-07 true resid norm 3.698257725532e-08 ||Ae||/||Ax|| 1.395812306629e-12 0 KSP preconditioned resid norm 3.809949813796e+06 true resid norm 3.702109838396e+08 ||Ae||/||Ax|| 3.702109838396e+08 60 KSP preconditioned resid norm 5.928602983876e-01 true resid norm 5.928602983876e-01 ||Ae||/||Ax|| 8.923275008553e-01 0 KSP preconditioned resid norm 5.179438201612e-01 true resid norm 3.250710514971e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.179438201612e-01 true resid norm 3.250710514971e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.966760428987e-14 true resid norm 7.545966269105e-14 ||Ae||/||Ax|| 2.321328286340e-14 1 KSP preconditioned resid norm 7.966760428987e-14 true resid norm 7.545966269105e-14 ||Ae||/||Ax|| 2.321328286340e-14 0 KSP preconditioned resid norm 1.033835105831e+00 true resid norm 2.054494906229e+02 ||Ae||/||Ax|| 2.054494906229e+02 1 KSP preconditioned resid norm 1.721270363501e-01 true resid norm 1.772266094130e+01 ||Ae||/||Ax|| 1.772266094130e+01 0 KSP preconditioned resid norm 2.287664874308e+02 true resid norm 3.141706116754e+01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.287664874308e+02 true resid norm 3.141706116754e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.263435537337e-11 true resid norm 1.930191711152e-11 ||Ae||/||Ax|| 6.143769147786e-13 1 KSP preconditioned resid norm 4.263435537337e-11 true resid norm 1.930191711152e-11 ||Ae||/||Ax|| 6.143769147786e-13 0 KSP preconditioned resid norm 3.937719734879e+02 true resid norm 1.874883066320e+05 ||Ae||/||Ax|| 1.874883066320e+05 61 KSP preconditioned resid norm 5.927978462698e-01 true resid norm 5.927978462698e-01 ||Ae||/||Ax|| 8.922335027542e-01 0 KSP preconditioned resid norm 2.012108955893e+01 true resid norm 1.407145543919e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.012108955893e+01 true resid norm 1.407145543919e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.941711699645e-12 true resid norm 2.950708049107e-12 ||Ae||/||Ax|| 2.096945878738e-12 1 KSP preconditioned resid norm 4.941711699645e-12 true resid norm 2.950708049107e-12 ||Ae||/||Ax|| 2.096945878738e-12 0 KSP preconditioned resid norm 3.155009698607e+01 true resid norm 2.954858506909e+04 ||Ae||/||Ax|| 2.954858506909e+04 0 KSP preconditioned resid norm 4.394139517734e+05 true resid norm 4.065913733480e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.394139517734e+05 true resid norm 4.065913733480e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.201387843927e-07 true resid norm 9.242186560010e-08 ||Ae||/||Ax|| 2.273089683115e-12 1 KSP preconditioned resid norm 1.201387843927e-07 true resid norm 9.242186560010e-08 ||Ae||/||Ax|| 2.273089683115e-12 0 KSP preconditioned resid norm 7.030059377776e+05 true resid norm 9.042247003330e+08 ||Ae||/||Ax|| 9.042247003330e+08 62 KSP preconditioned resid norm 5.919537264319e-01 true resid norm 5.919537264319e-01 ||Ae||/||Ax|| 8.909629988135e-01 0 KSP preconditioned resid norm 3.921276882243e+01 true resid norm 1.752866070607e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.921276882243e+01 true resid norm 1.752866070607e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.478780080025e-12 true resid norm 1.074682746094e-12 ||Ae||/||Ax|| 6.131003184529e-13 1 KSP preconditioned resid norm 7.478780080025e-12 true resid norm 1.074682746094e-12 ||Ae||/||Ax|| 6.131003184529e-13 0 KSP preconditioned resid norm 6.051435663370e+01 true resid norm 1.081789435855e+04 ||Ae||/||Ax|| 1.081789435855e+04 1 KSP preconditioned resid norm 8.475042638128e+00 true resid norm 8.648968591390e+02 ||Ae||/||Ax|| 8.648968591390e+02 0 KSP preconditioned resid norm 6.171310505682e+04 true resid norm 1.166624528674e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.171310505682e+04 true resid norm 1.166624528674e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.117035425221e-08 true resid norm 2.084284926060e-09 ||Ae||/||Ax|| 1.786594465341e-12 1 KSP preconditioned resid norm 1.117035425221e-08 true resid norm 2.084284926060e-09 ||Ae||/||Ax|| 1.786594465341e-12 0 KSP preconditioned resid norm 9.334852695260e+04 true resid norm 2.001338415085e+07 ||Ae||/||Ax|| 2.001338415085e+07 63 KSP preconditioned resid norm 5.918304743007e-01 true resid norm 5.918304743007e-01 ||Ae||/||Ax|| 8.907774892314e-01 0 KSP preconditioned resid norm 1.271757156211e+02 true resid norm 1.442068162039e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.271757156211e+02 true resid norm 1.442068162039e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.334657192625e-11 true resid norm 1.755668015052e-12 ||Ae||/||Ax|| 1.217465346832e-12 1 KSP preconditioned resid norm 2.334657192625e-11 true resid norm 1.755668015052e-12 ||Ae||/||Ax|| 1.217465346832e-12 0 KSP preconditioned resid norm 1.907435869396e+02 true resid norm 1.848711923036e+04 ||Ae||/||Ax|| 1.848711923036e+04 0 KSP preconditioned resid norm 2.463163205707e+06 true resid norm 2.564497576568e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.463163205707e+06 true resid norm 2.564497576568e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.347566836521e-07 true resid norm 3.700016154754e-08 ||Ae||/||Ax|| 1.442784032460e-12 1 KSP preconditioned resid norm 4.347566836521e-07 true resid norm 3.700016154754e-08 ||Ae||/||Ax|| 1.442784032460e-12 0 KSP preconditioned resid norm 3.698822458564e+06 true resid norm 3.607966636266e+08 ||Ae||/||Ax|| 3.607966636266e+08 64 KSP preconditioned resid norm 5.916348688499e-01 true resid norm 5.916348688499e-01 ||Ae||/||Ax|| 8.904830790245e-01 0 KSP preconditioned resid norm 9.178083267510e+00 true resid norm 1.576994924133e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.178083267510e+00 true resid norm 1.576994924133e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.122102451871e-12 true resid norm 2.576063074067e-12 ||Ae||/||Ax|| 1.633526547642e-12 1 KSP preconditioned resid norm 5.122102451871e-12 true resid norm 2.576063074067e-12 ||Ae||/||Ax|| 1.633526547642e-12 0 KSP preconditioned resid norm 1.564902268937e+01 true resid norm 2.210478804214e+04 ||Ae||/||Ax|| 2.210478804214e+04 1 KSP preconditioned resid norm 3.303401751858e+00 true resid norm 2.716649640785e+03 ||Ae||/||Ax|| 2.716649640785e+03 0 KSP preconditioned resid norm 2.703257356530e+04 true resid norm 3.707021829719e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.703257356530e+04 true resid norm 3.707021829719e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.306788053895e-08 true resid norm 9.205858705099e-09 ||Ae||/||Ax|| 2.483357025658e-12 1 KSP preconditioned resid norm 1.306788053895e-08 true resid norm 9.205858705099e-09 ||Ae||/||Ax|| 2.483357025658e-12 0 KSP preconditioned resid norm 4.675567763743e+04 true resid norm 8.110243296727e+07 ||Ae||/||Ax|| 8.110243296727e+07 65 KSP preconditioned resid norm 5.914431066657e-01 true resid norm 5.914431066657e-01 ||Ae||/||Ax|| 8.901944534056e-01 0 KSP preconditioned resid norm 1.292123063289e+01 true resid norm 1.579270302837e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.292123063289e+01 true resid norm 1.579270302837e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.710439570660e-12 true resid norm 2.548363112462e-12 ||Ae||/||Ax|| 1.613633276004e-12 1 KSP preconditioned resid norm 5.710439570660e-12 true resid norm 2.548363112462e-12 ||Ae||/||Ax|| 1.613633276004e-12 0 KSP preconditioned resid norm 2.079497685090e+01 true resid norm 2.178641699821e+04 ||Ae||/||Ax|| 2.178641699821e+04 1 KSP preconditioned resid norm 4.068151260640e+00 true resid norm 2.235880590500e+03 ||Ae||/||Ax|| 2.235880590500e+03 0 KSP preconditioned resid norm 2.936493457576e+04 true resid norm 3.013807321465e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.936493457576e+04 true resid norm 3.013807321465e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.654868770687e-08 true resid norm 7.565264176378e-09 ||Ae||/||Ax|| 2.510201671652e-12 1 KSP preconditioned resid norm 1.654868770687e-08 true resid norm 7.565264176378e-09 ||Ae||/||Ax|| 2.510201671652e-12 0 KSP preconditioned resid norm 4.809680205703e+04 true resid norm 6.585494986157e+07 ||Ae||/||Ax|| 6.585494986157e+07 66 KSP preconditioned resid norm 5.913378660812e-01 true resid norm 5.913378660812e-01 ||Ae||/||Ax|| 8.900360534115e-01 0 KSP preconditioned resid norm 8.055011517200e+01 true resid norm 1.481363614120e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.055011517200e+01 true resid norm 1.481363614120e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.157691046605e-11 true resid norm 2.337248942784e-12 ||Ae||/||Ax|| 1.577768564386e-12 1 KSP preconditioned resid norm 1.157691046605e-11 true resid norm 2.337248942784e-12 ||Ae||/||Ax|| 1.577768564386e-12 0 KSP preconditioned resid norm 1.194780984915e+02 true resid norm 2.291312903788e+04 ||Ae||/||Ax|| 2.291312903788e+04 0 KSP preconditioned resid norm 1.562577617260e+06 true resid norm 3.162548632421e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.562577617260e+06 true resid norm 3.162548632421e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.531197148422e-07 true resid norm 6.955139474877e-08 ||Ae||/||Ax|| 2.199219769643e-12 1 KSP preconditioned resid norm 2.531197148422e-07 true resid norm 6.955139474877e-08 ||Ae||/||Ax|| 2.199219769643e-12 0 KSP preconditioned resid norm 2.316893078437e+06 true resid norm 6.446392523306e+08 ||Ae||/||Ax|| 6.446392523306e+08 67 KSP preconditioned resid norm 5.911446104772e-01 true resid norm 5.911446104772e-01 ||Ae||/||Ax|| 8.897451800125e-01 0 KSP preconditioned resid norm 3.378924964593e+01 true resid norm 1.557514924799e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.378924964593e+01 true resid norm 1.557514924799e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.918960423550e-12 true resid norm 2.303848597703e-12 ||Ae||/||Ax|| 1.479182357113e-12 1 KSP preconditioned resid norm 5.918960423550e-12 true resid norm 2.303848597703e-12 ||Ae||/||Ax|| 1.479182357113e-12 0 KSP preconditioned resid norm 5.084440656493e+01 true resid norm 2.259304599402e+04 ||Ae||/||Ax|| 2.259304599402e+04 0 KSP preconditioned resid norm 6.557284297420e+05 true resid norm 3.118004113197e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.557284297420e+05 true resid norm 3.118004113197e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.437397587258e-07 true resid norm 7.205948047578e-08 ||Ae||/||Ax|| 2.311077146139e-12 1 KSP preconditioned resid norm 1.437397587258e-07 true resid norm 7.205948047578e-08 ||Ae||/||Ax|| 2.311077146139e-12 0 KSP preconditioned resid norm 9.823883172453e+05 true resid norm 6.799097454264e+08 ||Ae||/||Ax|| 6.799097454264e+08 68 KSP preconditioned resid norm 5.909708050484e-01 true resid norm 5.909708050484e-01 ||Ae||/||Ax|| 8.894835815139e-01 0 KSP preconditioned resid norm 1.029814855951e+02 true resid norm 1.528108053546e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.029814855951e+02 true resid norm 1.528108053546e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.441035778527e-11 true resid norm 1.764200363703e-12 ||Ae||/||Ax|| 1.154499748633e-12 1 KSP preconditioned resid norm 1.441035778527e-11 true resid norm 1.764200363703e-12 ||Ae||/||Ax|| 1.154499748633e-12 0 KSP preconditioned resid norm 1.525231162412e+02 true resid norm 1.723355688473e+04 ||Ae||/||Ax|| 1.723355688473e+04 0 KSP preconditioned resid norm 1.991893423870e+06 true resid norm 2.392943171797e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.991893423870e+06 true resid norm 2.392943171797e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.957228856265e-07 true resid norm 4.096332740568e-08 ||Ae||/||Ax|| 1.711838705092e-12 1 KSP preconditioned resid norm 2.957228856265e-07 true resid norm 4.096332740568e-08 ||Ae||/||Ax|| 1.711838705092e-12 0 KSP preconditioned resid norm 2.948628795930e+06 true resid norm 3.878356440933e+08 ||Ae||/||Ax|| 3.878356440933e+08 69 KSP preconditioned resid norm 5.896448468440e-01 true resid norm 5.896448468440e-01 ||Ae||/||Ax|| 8.874878517038e-01 0 KSP preconditioned resid norm 9.959789410450e+00 true resid norm 1.824191731213e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.959789410450e+00 true resid norm 1.824191731213e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.971054140591e-12 true resid norm 2.985849351125e-13 ||Ae||/||Ax|| 1.636806756677e-13 1 KSP preconditioned resid norm 1.971054140591e-12 true resid norm 2.985849351125e-13 ||Ae||/||Ax|| 1.636806756677e-13 0 KSP preconditioned resid norm 1.726064485951e+01 true resid norm 3.573429631643e+03 ||Ae||/||Ax|| 3.573429631643e+03 1 KSP preconditioned resid norm 1.767181500542e+00 true resid norm 1.664467395793e+02 ||Ae||/||Ax|| 1.664467395793e+02 0 KSP preconditioned resid norm 3.163790325043e+03 true resid norm 2.324997666159e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.163790325043e+03 true resid norm 2.324997666159e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.289145898962e-10 true resid norm 7.670345105221e-11 ||Ae||/||Ax|| 3.299076475158e-13 1 KSP preconditioned resid norm 6.289145898962e-10 true resid norm 7.670345105221e-11 ||Ae||/||Ax|| 3.299076475158e-13 0 KSP preconditioned resid norm 4.864956911035e+03 true resid norm 8.021173018370e+05 ||Ae||/||Ax|| 8.021173018370e+05 70 KSP preconditioned resid norm 5.889057970138e-01 true resid norm 5.889057970138e-01 ||Ae||/||Ax|| 8.863754910182e-01 0 KSP preconditioned resid norm 1.324284912711e+01 true resid norm 1.610385052874e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.324284912711e+01 true resid norm 1.610385052874e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.041728266896e-12 true resid norm 2.112113869715e-12 ||Ae||/||Ax|| 1.311558292189e-12 1 KSP preconditioned resid norm 4.041728266896e-12 true resid norm 2.112113869715e-12 ||Ae||/||Ax|| 1.311558292189e-12 0 KSP preconditioned resid norm 2.102017771354e+01 true resid norm 1.998307524113e+04 ||Ae||/||Ax|| 1.998307524113e+04 1 KSP preconditioned resid norm 3.260579450742e+00 true resid norm 1.564822546570e+03 ||Ae||/||Ax|| 1.564822546570e+03 0 KSP preconditioned resid norm 2.226546285877e+04 true resid norm 2.098460913561e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.226546285877e+04 true resid norm 2.098460913561e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.188599597040e-09 true resid norm 4.612612176875e-09 ||Ae||/||Ax|| 2.198092967597e-12 1 KSP preconditioned resid norm 7.188599597040e-09 true resid norm 4.612612176875e-09 ||Ae||/||Ax|| 2.198092967597e-12 0 KSP preconditioned resid norm 3.635036633697e+04 true resid norm 4.533513630815e+07 ||Ae||/||Ax|| 4.533513630815e+07 71 KSP preconditioned resid norm 5.883704232756e-01 true resid norm 5.883704232756e-01 ||Ae||/||Ax|| 8.855696878447e-01 0 KSP preconditioned resid norm 7.920074111172e+00 true resid norm 1.547320913348e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.920074111172e+00 true resid norm 1.547320913348e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.316080647221e-12 true resid norm 2.537037818142e-12 ||Ae||/||Ax|| 1.639632603848e-12 1 KSP preconditioned resid norm 3.316080647221e-12 true resid norm 2.537037818142e-12 ||Ae||/||Ax|| 1.639632603848e-12 0 KSP preconditioned resid norm 1.442399388621e+01 true resid norm 2.368516623298e+04 ||Ae||/||Ax|| 2.368516623298e+04 1 KSP preconditioned resid norm 3.086776130929e+00 true resid norm 3.332271030731e+03 ||Ae||/||Ax|| 3.332271030731e+03 0 KSP preconditioned resid norm 2.999510096744e+04 true resid norm 4.545338210712e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.999510096744e+04 true resid norm 4.545338210712e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.577628756679e-09 true resid norm 1.057139225253e-08 ||Ae||/||Ax|| 2.325765820377e-12 1 KSP preconditioned resid norm 5.577628756679e-09 true resid norm 1.057139225253e-08 ||Ae||/||Ax|| 2.325765820377e-12 0 KSP preconditioned resid norm 5.577865132754e+04 true resid norm 1.003633683157e+08 ||Ae||/||Ax|| 1.003633683157e+08 72 KSP preconditioned resid norm 5.882348116387e-01 true resid norm 5.882348116387e-01 ||Ae||/||Ax|| 8.853655756899e-01 0 KSP preconditioned resid norm 1.092002953913e+02 true resid norm 1.442811838005e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.092002953913e+02 true resid norm 1.442811838005e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.510142363396e-11 true resid norm 2.472805557534e-12 ||Ae||/||Ax|| 1.713879448725e-12 1 KSP preconditioned resid norm 1.510142363396e-11 true resid norm 2.472805557534e-12 ||Ae||/||Ax|| 1.713879448725e-12 0 KSP preconditioned resid norm 1.617555465975e+02 true resid norm 2.174335767635e+04 ||Ae||/||Ax|| 2.174335767635e+04 0 KSP preconditioned resid norm 2.117461285795e+06 true resid norm 3.004256903182e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.117461285795e+06 true resid norm 3.004256903182e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.784290598771e-07 true resid norm 6.296997145031e-08 ||Ae||/||Ax|| 2.096024856717e-12 1 KSP preconditioned resid norm 2.784290598771e-07 true resid norm 6.296997145031e-08 ||Ae||/||Ax|| 2.096024856717e-12 0 KSP preconditioned resid norm 3.138421687462e+06 true resid norm 5.514384235017e+08 ||Ae||/||Ax|| 5.514384235017e+08 73 KSP preconditioned resid norm 5.879913320737e-01 true resid norm 5.879913320737e-01 ||Ae||/||Ax|| 8.849991090664e-01 0 KSP preconditioned resid norm 3.904908866645e+01 true resid norm 1.513472973354e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.904908866645e+01 true resid norm 1.513472973354e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.170949789843e-12 true resid norm 3.071966572315e-12 ||Ae||/||Ax|| 2.029746567266e-12 1 KSP preconditioned resid norm 8.170949789843e-12 true resid norm 3.071966572315e-12 ||Ae||/||Ax|| 2.029746567266e-12 0 KSP preconditioned resid norm 5.862913141809e+01 true resid norm 2.549874294023e+04 ||Ae||/||Ax|| 2.549874294023e+04 0 KSP preconditioned resid norm 7.707267786027e+05 true resid norm 3.513193195809e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.707267786027e+05 true resid norm 3.513193195809e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.957443610758e-07 true resid norm 8.973046683565e-08 ||Ae||/||Ax|| 2.554099983533e-12 1 KSP preconditioned resid norm 1.957443610758e-07 true resid norm 8.973046683565e-08 ||Ae||/||Ax|| 2.554099983533e-12 0 KSP preconditioned resid norm 1.168625078742e+06 true resid norm 7.695198083471e+08 ||Ae||/||Ax|| 7.695198083471e+08 74 KSP preconditioned resid norm 5.874271286263e-01 true resid norm 5.874271286262e-01 ||Ae||/||Ax|| 8.841499136428e-01 0 KSP preconditioned resid norm 6.814273537836e+01 true resid norm 1.685650475236e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.814273537836e+01 true resid norm 1.685650475236e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.630067001308e-12 true resid norm 1.639428530065e-12 ||Ae||/||Ax|| 9.725791640377e-13 1 KSP preconditioned resid norm 8.630067001308e-12 true resid norm 1.639428530065e-12 ||Ae||/||Ax|| 9.725791640377e-13 0 KSP preconditioned resid norm 1.012230599537e+02 true resid norm 1.439070776509e+04 ||Ae||/||Ax|| 1.439070776509e+04 0 KSP preconditioned resid norm 1.313576699279e+06 true resid norm 2.005797728971e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.313576699279e+06 true resid norm 2.005797728971e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.929798598126e-07 true resid norm 4.228044967898e-08 ||Ae||/||Ax|| 2.107911932908e-12 1 KSP preconditioned resid norm 1.929798598126e-07 true resid norm 4.228044967898e-08 ||Ae||/||Ax|| 2.107911932908e-12 0 KSP preconditioned resid norm 1.947249912005e+06 true resid norm 3.651807360488e+08 ||Ae||/||Ax|| 3.651807360488e+08 75 KSP preconditioned resid norm 5.868568962526e-01 true resid norm 5.868568962526e-01 ||Ae||/||Ax|| 8.832916439454e-01 0 KSP preconditioned resid norm 7.973593956915e+00 true resid norm 1.815468893600e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.973593956915e+00 true resid norm 1.815468893600e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.459684326756e-12 true resid norm 3.431243764136e-13 ||Ae||/||Ax|| 1.890004161587e-13 1 KSP preconditioned resid norm 2.459684326756e-12 true resid norm 3.431243764136e-13 ||Ae||/||Ax|| 1.890004161587e-13 0 KSP preconditioned resid norm 1.451784367052e+01 true resid norm 4.216755428875e+03 ||Ae||/||Ax|| 4.216755428875e+03 1 KSP preconditioned resid norm 2.272817824119e+00 true resid norm 3.227785703171e+02 ||Ae||/||Ax|| 3.227785703171e+02 0 KSP preconditioned resid norm 4.387520111849e+03 true resid norm 4.916672293340e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.387520111849e+03 true resid norm 4.916672293340e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.228893771290e-09 true resid norm 1.591744413248e-10 ||Ae||/||Ax|| 3.237442559277e-13 1 KSP preconditioned resid norm 1.228893771290e-09 true resid norm 1.591744413248e-10 ||Ae||/||Ax|| 3.237442559277e-13 0 KSP preconditioned resid norm 7.014386174558e+03 true resid norm 1.590707000623e+06 ||Ae||/||Ax|| 1.590707000623e+06 76 KSP preconditioned resid norm 5.864843728836e-01 true resid norm 5.864843728836e-01 ||Ae||/||Ax|| 8.827309505615e-01 0 KSP preconditioned resid norm 6.250747246117e+01 true resid norm 1.680360161727e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.250747246117e+01 true resid norm 1.680360161727e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.024294628567e-11 true resid norm 1.496340330347e-12 ||Ae||/||Ax|| 8.904878635119e-13 1 KSP preconditioned resid norm 1.024294628567e-11 true resid norm 1.496340330347e-12 ||Ae||/||Ax|| 8.904878635119e-13 0 KSP preconditioned resid norm 9.290074177060e+01 true resid norm 1.431458510242e+04 ||Ae||/||Ax|| 1.431458510242e+04 0 KSP preconditioned resid norm 1.209934875804e+06 true resid norm 2.002398243896e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.209934875804e+06 true resid norm 2.002398243896e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.927413143647e-07 true resid norm 3.890279585053e-08 ||Ae||/||Ax|| 1.942810126263e-12 1 KSP preconditioned resid norm 1.927413143647e-07 true resid norm 3.890279585053e-08 ||Ae||/||Ax|| 1.942810126263e-12 0 KSP preconditioned resid norm 1.795157337590e+06 true resid norm 3.681390335541e+08 ||Ae||/||Ax|| 3.681390335541e+08 77 KSP preconditioned resid norm 5.863438456782e-01 true resid norm 5.863438456782e-01 ||Ae||/||Ax|| 8.825194398728e-01 0 KSP preconditioned resid norm 3.557457154005e+01 true resid norm 1.495175850272e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.557457154005e+01 true resid norm 1.495175850272e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.552976352371e-12 true resid norm 2.697118522673e-12 ||Ae||/||Ax|| 1.803880474783e-12 1 KSP preconditioned resid norm 6.552976352371e-12 true resid norm 2.697118522673e-12 ||Ae||/||Ax|| 1.803880474783e-12 0 KSP preconditioned resid norm 5.314014688463e+01 true resid norm 2.602314726827e+04 ||Ae||/||Ax|| 2.602314726827e+04 0 KSP preconditioned resid norm 7.104613432327e+05 true resid norm 3.586135159711e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.104613432327e+05 true resid norm 3.586135159711e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.700579336387e-07 true resid norm 8.241198236784e-08 ||Ae||/||Ax|| 2.298072400999e-12 1 KSP preconditioned resid norm 1.700579336387e-07 true resid norm 8.241198236784e-08 ||Ae||/||Ax|| 2.298072400999e-12 0 KSP preconditioned resid norm 1.070349184999e+06 true resid norm 7.869520417219e+08 ||Ae||/||Ax|| 7.869520417219e+08 78 KSP preconditioned resid norm 5.862761416078e-01 true resid norm 5.862761416078e-01 ||Ae||/||Ax|| 8.824175369387e-01 0 KSP preconditioned resid norm 1.137370859163e+02 true resid norm 1.431251994420e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.137370859163e+02 true resid norm 1.431251994420e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.550599539711e-11 true resid norm 2.140123663840e-12 ||Ae||/||Ax|| 1.495280825588e-12 1 KSP preconditioned resid norm 1.550599539711e-11 true resid norm 2.140123663840e-12 ||Ae||/||Ax|| 1.495280825588e-12 0 KSP preconditioned resid norm 1.685511344030e+02 true resid norm 2.128989830652e+04 ||Ae||/||Ax|| 2.128989830652e+04 0 KSP preconditioned resid norm 2.205305922755e+06 true resid norm 2.943649269335e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.205305922755e+06 true resid norm 2.943649269335e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.001524327934e-07 true resid norm 5.345386935089e-08 ||Ae||/||Ax|| 1.815904833084e-12 1 KSP preconditioned resid norm 3.001524327934e-07 true resid norm 5.345386935089e-08 ||Ae||/||Ax|| 1.815904833084e-12 0 KSP preconditioned resid norm 3.269519691203e+06 true resid norm 5.226429163913e+08 ||Ae||/||Ax|| 5.226429163913e+08 79 KSP preconditioned resid norm 5.853325048441e-01 true resid norm 5.853325048441e-01 ||Ae||/||Ax|| 8.809972478126e-01 0 KSP preconditioned resid norm 1.285183313505e+01 true resid norm 1.902522964885e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.285183313505e+01 true resid norm 1.902522964885e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.400674526664e-12 true resid norm 3.109052139726e-13 ||Ae||/||Ax|| 1.634173251577e-13 1 KSP preconditioned resid norm 3.400674526664e-12 true resid norm 3.109052139726e-13 ||Ae||/||Ax|| 1.634173251577e-13 0 KSP preconditioned resid norm 2.224990077161e+01 true resid norm 3.354904043059e+03 ||Ae||/||Ax|| 3.354904043059e+03 1 KSP preconditioned resid norm 3.540335796102e+00 true resid norm 3.236675479086e+02 ||Ae||/||Ax|| 3.236675479086e+02 0 KSP preconditioned resid norm 1.210770398169e+04 true resid norm 5.113311265897e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.210770398169e+04 true resid norm 5.113311265897e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.382186944719e-09 true resid norm 2.286802654925e-10 ||Ae||/||Ax|| 4.472253958364e-13 1 KSP preconditioned resid norm 2.382186944719e-09 true resid norm 2.286802654925e-10 ||Ae||/||Ax|| 4.472253958364e-13 0 KSP preconditioned resid norm 1.854289484283e+04 true resid norm 2.136959299308e+06 ||Ae||/||Ax|| 2.136959299308e+06 80 KSP preconditioned resid norm 5.845194027528e-01 true resid norm 5.845194027528e-01 ||Ae||/||Ax|| 8.797734293868e-01 0 KSP preconditioned resid norm 5.129824230359e+01 true resid norm 1.787548726169e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.129824230359e+01 true resid norm 1.787548726169e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.352487657917e-12 true resid norm 1.098422290421e-12 ||Ae||/||Ax|| 6.144852301593e-13 1 KSP preconditioned resid norm 7.352487657917e-12 true resid norm 1.098422290421e-12 ||Ae||/||Ax|| 6.144852301593e-13 0 KSP preconditioned resid norm 7.752428823074e+01 true resid norm 9.082631362949e+03 ||Ae||/||Ax|| 9.082631362949e+03 0 KSP preconditioned resid norm 9.791629346963e+05 true resid norm 1.290809980827e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.791629346963e+05 true resid norm 1.290809980827e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.287113113167e-07 true resid norm 2.046236012232e-08 ||Ae||/||Ax|| 1.585234110850e-12 1 KSP preconditioned resid norm 1.287113113167e-07 true resid norm 2.046236012232e-08 ||Ae||/||Ax|| 1.585234110850e-12 0 KSP preconditioned resid norm 1.463510519927e+06 true resid norm 1.923865562740e+08 ||Ae||/||Ax|| 1.923865562740e+08 81 KSP preconditioned resid norm 5.843606398102e-01 true resid norm 5.843606398102e-01 ||Ae||/||Ax|| 8.795344716759e-01 0 KSP preconditioned resid norm 8.540929220162e+01 true resid norm 1.460153032186e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.540929220162e+01 true resid norm 1.460153032186e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.251223185764e-11 true resid norm 2.700840277944e-12 ||Ae||/||Ax|| 1.849696722473e-12 1 KSP preconditioned resid norm 1.251223185764e-11 true resid norm 2.700840277944e-12 ||Ae||/||Ax|| 1.849696722473e-12 0 KSP preconditioned resid norm 1.273080251685e+02 true resid norm 2.428337899099e+04 ||Ae||/||Ax|| 2.428337899099e+04 0 KSP preconditioned resid norm 1.659645652487e+06 true resid norm 3.348400350969e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.659645652487e+06 true resid norm 3.348400350969e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.372997882797e-07 true resid norm 7.347016639941e-08 ||Ae||/||Ax|| 2.194187035554e-12 1 KSP preconditioned resid norm 2.372997882797e-07 true resid norm 7.347016639941e-08 ||Ae||/||Ax|| 2.194187035554e-12 0 KSP preconditioned resid norm 2.480495543944e+06 true resid norm 6.845724150934e+08 ||Ae||/||Ax|| 6.845724150934e+08 82 KSP preconditioned resid norm 5.842984450525e-01 true resid norm 5.842984450525e-01 ||Ae||/||Ax|| 8.794408609335e-01 0 KSP preconditioned resid norm 8.992052614542e+01 true resid norm 1.408770938845e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.992052614542e+01 true resid norm 1.408770938845e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.123782548146e-11 true resid norm 2.665039187047e-12 ||Ae||/||Ax|| 1.891747702599e-12 1 KSP preconditioned resid norm 1.123782548146e-11 true resid norm 2.665039187047e-12 ||Ae||/||Ax|| 1.891747702599e-12 0 KSP preconditioned resid norm 1.341038910343e+02 true resid norm 2.553689722159e+04 ||Ae||/||Ax|| 2.553689722159e+04 0 KSP preconditioned resid norm 1.748515981060e+06 true resid norm 3.519698125828e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.748515981060e+06 true resid norm 3.519698125828e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.205306864500e-07 true resid norm 7.778023791760e-08 ||Ae||/||Ax|| 2.209855366483e-12 1 KSP preconditioned resid norm 2.205306864500e-07 true resid norm 7.778023791760e-08 ||Ae||/||Ax|| 2.209855366483e-12 0 KSP preconditioned resid norm 2.613685337068e+06 true resid norm 7.205543009626e+08 ||Ae||/||Ax|| 7.205543009626e+08 83 KSP preconditioned resid norm 5.836925298371e-01 true resid norm 5.836925298370e-01 ||Ae||/||Ax|| 8.785288841805e-01 0 KSP preconditioned resid norm 1.566216522481e+01 true resid norm 1.762689629179e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.566216522481e+01 true resid norm 1.762689629179e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.474200242271e-12 true resid norm 5.672372200220e-13 ||Ae||/||Ax|| 3.218020975628e-13 1 KSP preconditioned resid norm 2.474200242271e-12 true resid norm 5.672372200220e-13 ||Ae||/||Ax|| 3.218020975628e-13 0 KSP preconditioned resid norm 2.476535413970e+01 true resid norm 5.179236647466e+03 ||Ae||/||Ax|| 5.179236647466e+03 1 KSP preconditioned resid norm 3.886472467620e+00 true resid norm 4.607747242004e+02 ||Ae||/||Ax|| 4.607747242004e+02 0 KSP preconditioned resid norm 2.642989888257e+04 true resid norm 6.691168916981e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.642989888257e+04 true resid norm 6.691168916981e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.451274694668e-09 true resid norm 5.548097328087e-10 ||Ae||/||Ax|| 8.291671301268e-13 1 KSP preconditioned resid norm 3.451274694668e-09 true resid norm 5.548097328087e-10 ||Ae||/||Ax|| 8.291671301268e-13 0 KSP preconditioned resid norm 3.978749399029e+04 true resid norm 5.239781741328e+06 ||Ae||/||Ax|| 5.239781741328e+06 84 KSP preconditioned resid norm 5.833447809338e-01 true resid norm 5.833447809337e-01 ||Ae||/||Ax|| 8.780054794075e-01 0 KSP preconditioned resid norm 1.579817072976e+01 true resid norm 1.847120603885e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.579817072976e+01 true resid norm 1.847120603885e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.949880851394e-12 true resid norm 4.536574430896e-13 ||Ae||/||Ax|| 2.456025026927e-13 1 KSP preconditioned resid norm 1.949880851394e-12 true resid norm 4.536574430896e-13 ||Ae||/||Ax|| 2.456025026927e-13 0 KSP preconditioned resid norm 2.608767270889e+01 true resid norm 4.616421797210e+03 ||Ae||/||Ax|| 4.616421797210e+03 1 KSP preconditioned resid norm 2.834914979831e+00 true resid norm 2.374949050100e+02 ||Ae||/||Ax|| 2.374949050100e+02 0 KSP preconditioned resid norm 9.577533214753e+03 true resid norm 3.434071579533e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.577533214753e+03 true resid norm 3.434071579533e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.042397070877e-09 true resid norm 4.088972236920e-10 ||Ae||/||Ax|| 1.190706757917e-12 1 KSP preconditioned resid norm 1.042397070877e-09 true resid norm 4.088972236920e-10 ||Ae||/||Ax|| 1.190706757917e-12 0 KSP preconditioned resid norm 1.437880068149e+04 true resid norm 3.751388111097e+06 ||Ae||/||Ax|| 3.751388111097e+06 85 KSP preconditioned resid norm 5.833349314379e-01 true resid norm 5.833349314378e-01 ||Ae||/||Ax|| 8.779906547075e-01 0 KSP preconditioned resid norm 4.750562102904e+01 true resid norm 1.386915909260e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.750562102904e+01 true resid norm 1.386915909260e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.134517663227e-12 true resid norm 3.169630514519e-12 ||Ae||/||Ax|| 2.285380456997e-12 1 KSP preconditioned resid norm 7.134517663227e-12 true resid norm 3.169630514519e-12 ||Ae||/||Ax|| 2.285380456997e-12 0 KSP preconditioned resid norm 7.055485444445e+01 true resid norm 2.929761031751e+04 ||Ae||/||Ax|| 2.929761031751e+04 0 KSP preconditioned resid norm 9.419265122794e+05 true resid norm 4.031732428306e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.419265122794e+05 true resid norm 4.031732428306e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.985240298731e-07 true resid norm 9.672100610764e-08 ||Ae||/||Ax|| 2.398993678960e-12 1 KSP preconditioned resid norm 1.985240298731e-07 true resid norm 9.672100610764e-08 ||Ae||/||Ax|| 2.398993678960e-12 0 KSP preconditioned resid norm 1.412581656802e+06 true resid norm 8.846053470960e+08 ||Ae||/||Ax|| 8.846053470960e+08 86 KSP preconditioned resid norm 5.833010574132e-01 true resid norm 5.833010574131e-01 ||Ae||/||Ax|| 8.779396701436e-01 0 KSP preconditioned resid norm 1.192404334527e+02 true resid norm 1.425183123674e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.192404334527e+02 true resid norm 1.425183123674e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.235604647787e-11 true resid norm 2.012951294373e-12 ||Ae||/||Ax|| 1.412415893042e-12 1 KSP preconditioned resid norm 1.235604647787e-11 true resid norm 2.012951294373e-12 ||Ae||/||Ax|| 1.412415893042e-12 0 KSP preconditioned resid norm 1.758432326455e+02 true resid norm 2.042921119685e+04 ||Ae||/||Ax|| 2.042921119685e+04 0 KSP preconditioned resid norm 2.311016294074e+06 true resid norm 2.827058116366e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.311016294074e+06 true resid norm 2.827058116366e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.068968815656e-07 true resid norm 4.919920936275e-08 ||Ae||/||Ax|| 1.740297062800e-12 1 KSP preconditioned resid norm 2.068968815656e-07 true resid norm 4.919920936275e-08 ||Ae||/||Ax|| 1.740297062800e-12 0 KSP preconditioned resid norm 3.407232359321e+06 true resid norm 4.735317430036e+08 ||Ae||/||Ax|| 4.735317430036e+08 87 KSP preconditioned resid norm 5.829989969913e-01 true resid norm 5.829989969912e-01 ||Ae||/||Ax|| 8.774850321418e-01 0 KSP preconditioned resid norm 3.998747085072e+01 true resid norm 1.762721503673e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.998747085072e+01 true resid norm 1.762721503673e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.166969562084e-12 true resid norm 6.486870653528e-13 ||Ae||/||Ax|| 3.680031496758e-13 1 KSP preconditioned resid norm 3.166969562084e-12 true resid norm 6.486870653528e-13 ||Ae||/||Ax|| 3.680031496758e-13 0 KSP preconditioned resid norm 5.915011054672e+01 true resid norm 7.133627477421e+03 ||Ae||/||Ax|| 7.133627477421e+03 0 KSP preconditioned resid norm 7.627346460647e+05 true resid norm 1.056638572840e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.627346460647e+05 true resid norm 1.056638572840e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.206469550246e-08 true resid norm 1.123586577536e-08 ||Ae||/||Ax|| 1.063359417702e-12 1 KSP preconditioned resid norm 6.206469550246e-08 true resid norm 1.123586577536e-08 ||Ae||/||Ax|| 1.063359417702e-12 0 KSP preconditioned resid norm 1.123499573231e+06 true resid norm 1.189736380226e+08 ||Ae||/||Ax|| 1.189736380226e+08 88 KSP preconditioned resid norm 5.824042251653e-01 true resid norm 5.824042251652e-01 ||Ae||/||Ax|| 8.765898275573e-01 0 KSP preconditioned resid norm 6.971219401195e+00 true resid norm 1.807720222312e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.971219401195e+00 true resid norm 1.807720222312e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.247867208892e-12 true resid norm 3.789951794609e-13 ||Ae||/||Ax|| 2.096536702876e-13 1 KSP preconditioned resid norm 1.247867208892e-12 true resid norm 3.789951794609e-13 ||Ae||/||Ax|| 2.096536702876e-13 0 KSP preconditioned resid norm 1.252183439206e+01 true resid norm 4.360337135947e+03 ||Ae||/||Ax|| 4.360337135947e+03 1 KSP preconditioned resid norm 2.103328754386e+00 true resid norm 3.614838050848e+02 ||Ae||/||Ax|| 3.614838050848e+02 0 KSP preconditioned resid norm 6.979189452907e+03 true resid norm 5.474992312237e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.979189452907e+03 true resid norm 5.474992312237e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.173120402766e-10 true resid norm 1.907318411549e-10 ||Ae||/||Ax|| 3.483691488088e-13 1 KSP preconditioned resid norm 9.173120402766e-10 true resid norm 1.907318411549e-10 ||Ae||/||Ax|| 3.483691488088e-13 0 KSP preconditioned resid norm 1.055666429308e+04 true resid norm 1.946817034359e+06 ||Ae||/||Ax|| 1.946817034359e+06 89 KSP preconditioned resid norm 5.819404872683e-01 true resid norm 5.819404872682e-01 ||Ae||/||Ax|| 8.758918451157e-01 0 KSP preconditioned resid norm 1.171719403375e+02 true resid norm 1.533726361322e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.171719403375e+02 true resid norm 1.533726361322e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.507753137557e-11 true resid norm 1.656754525441e-12 ||Ae||/||Ax|| 1.080215198240e-12 1 KSP preconditioned resid norm 1.507753137557e-11 true resid norm 1.656754525441e-12 ||Ae||/||Ax|| 1.080215198240e-12 0 KSP preconditioned resid norm 1.713834710495e+02 true resid norm 1.694232906255e+04 ||Ae||/||Ax|| 1.694232906255e+04 0 KSP preconditioned resid norm 2.267713864968e+06 true resid norm 2.352031145597e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.267713864968e+06 true resid norm 2.352031145597e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.483320177602e-07 true resid norm 3.371772002715e-08 ||Ae||/||Ax|| 1.433557548346e-12 1 KSP preconditioned resid norm 2.483320177602e-07 true resid norm 3.371772002715e-08 ||Ae||/||Ax|| 1.433557548346e-12 0 KSP preconditioned resid norm 3.314656971024e+06 true resid norm 3.271599665189e+08 ||Ae||/||Ax|| 3.271599665189e+08 90 KSP preconditioned resid norm 5.805277469629e-01 true resid norm 5.805277469629e-01 ||Ae||/||Ax|| 8.737654975942e-01 0 KSP preconditioned resid norm 7.556994708818e-01 true resid norm 3.238265285333e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.556994708818e-01 true resid norm 3.238265285333e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.403845620262e-13 true resid norm 9.384504348959e-14 ||Ae||/||Ax|| 2.898003567362e-14 1 KSP preconditioned resid norm 1.403845620262e-13 true resid norm 9.384504348959e-14 ||Ae||/||Ax|| 2.898003567362e-14 0 KSP preconditioned resid norm 1.431048996739e+00 true resid norm 2.305182412482e+02 ||Ae||/||Ax|| 2.305182412482e+02 1 KSP preconditioned resid norm 1.820810784106e-01 true resid norm 1.341964601131e+01 ||Ae||/||Ax|| 1.341964601131e+01 0 KSP preconditioned resid norm 2.143023034754e+02 true resid norm 2.210114040080e+01 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.143023034754e+02 true resid norm 2.210114040080e+01 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.780027858452e-11 true resid norm 1.104665955924e-11 ||Ae||/||Ax|| 4.998230570418e-13 1 KSP preconditioned resid norm 3.780027858452e-11 true resid norm 1.104665955924e-11 ||Ae||/||Ax|| 4.998230570418e-13 0 KSP preconditioned resid norm 3.504567738402e+02 true resid norm 1.093748627228e+05 ||Ae||/||Ax|| 1.093748627228e+05 91 KSP preconditioned resid norm 5.804055078487e-01 true resid norm 5.804055078487e-01 ||Ae||/||Ax|| 8.735815127270e-01 0 KSP preconditioned resid norm 3.158687200419e+01 true resid norm 1.416104290344e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.158687200419e+01 true resid norm 1.416104290344e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.244679655833e-12 true resid norm 2.963175603927e-12 ||Ae||/||Ax|| 2.092484024046e-12 1 KSP preconditioned resid norm 6.244679655833e-12 true resid norm 2.963175603927e-12 ||Ae||/||Ax|| 2.092484024046e-12 0 KSP preconditioned resid norm 4.798514922521e+01 true resid norm 2.887862428400e+04 ||Ae||/||Ax|| 2.887862428400e+04 0 KSP preconditioned resid norm 6.394308191634e+05 true resid norm 3.974417995165e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.394308191634e+05 true resid norm 3.974417995165e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.562537743433e-07 true resid norm 9.185447772751e-08 ||Ae||/||Ax|| 2.311142860143e-12 1 KSP preconditioned resid norm 1.562537743433e-07 true resid norm 9.185447772751e-08 ||Ae||/||Ax|| 2.311142860143e-12 0 KSP preconditioned resid norm 9.822860946702e+05 true resid norm 8.795690602290e+08 ||Ae||/||Ax|| 8.795690602290e+08 92 KSP preconditioned resid norm 5.793523053958e-01 true resid norm 5.793523053958e-01 ||Ae||/||Ax|| 8.719963137935e-01 0 KSP preconditioned resid norm 5.111607574546e+01 true resid norm 1.694029865489e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.111607574546e+01 true resid norm 1.694029865489e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.869362598310e-12 true resid norm 1.395781692390e-12 ||Ae||/||Ax|| 8.239416086019e-13 1 KSP preconditioned resid norm 8.869362598310e-12 true resid norm 1.395781692390e-12 ||Ae||/||Ax|| 8.239416086019e-13 0 KSP preconditioned resid norm 7.747343073638e+01 true resid norm 1.350834865747e+04 ||Ae||/||Ax|| 1.350834865747e+04 0 KSP preconditioned resid norm 9.646228444529e+05 true resid norm 1.888679638275e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.646228444529e+05 true resid norm 1.888679638275e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.814486071819e-07 true resid norm 3.645899554871e-08 ||Ae||/||Ax|| 1.930395966042e-12 1 KSP preconditioned resid norm 1.814486071819e-07 true resid norm 3.645899554871e-08 ||Ae||/||Ax|| 1.930395966042e-12 0 KSP preconditioned resid norm 1.442939231682e+06 true resid norm 3.652889929148e+08 ||Ae||/||Ax|| 3.652889929148e+08 93 KSP preconditioned resid norm 5.790676556936e-01 true resid norm 5.790676556936e-01 ||Ae||/||Ax|| 8.715678810614e-01 0 KSP preconditioned resid norm 1.211809242713e+02 true resid norm 1.476259870633e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.211809242713e+02 true resid norm 1.476259870633e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.458475779350e-11 true resid norm 1.840603999110e-12 ||Ae||/||Ax|| 1.246802162495e-12 1 KSP preconditioned resid norm 2.458475779350e-11 true resid norm 1.840603999110e-12 ||Ae||/||Ax|| 1.246802162495e-12 0 KSP preconditioned resid norm 1.806135167741e+02 true resid norm 1.771464011215e+04 ||Ae||/||Ax|| 1.771464011215e+04 0 KSP preconditioned resid norm 2.344872063761e+06 true resid norm 2.459046505164e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.344872063761e+06 true resid norm 2.459046505164e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.005958017645e-07 true resid norm 3.524344338907e-08 ||Ae||/||Ax|| 1.433215814140e-12 1 KSP preconditioned resid norm 4.005958017645e-07 true resid norm 3.524344338907e-08 ||Ae||/||Ax|| 1.433215814140e-12 0 KSP preconditioned resid norm 3.497168058421e+06 true resid norm 3.475528395444e+08 ||Ae||/||Ax|| 3.475528395444e+08 94 KSP preconditioned resid norm 5.785720406840e-01 true resid norm 5.785720406840e-01 ||Ae||/||Ax|| 8.708219196535e-01 0 KSP preconditioned resid norm 8.335095456735e+00 true resid norm 1.789667043600e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.335095456735e+00 true resid norm 1.789667043600e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.560623267105e-12 true resid norm 3.930049335444e-13 ||Ae||/||Ax|| 2.195966757894e-13 1 KSP preconditioned resid norm 1.560623267105e-12 true resid norm 3.930049335444e-13 ||Ae||/||Ax|| 2.195966757894e-13 0 KSP preconditioned resid norm 1.532975782280e+01 true resid norm 4.611363927514e+03 ||Ae||/||Ax|| 4.611363927514e+03 1 KSP preconditioned resid norm 2.804788016722e+00 true resid norm 4.915315731076e+02 ||Ae||/||Ax|| 4.915315731076e+02 0 KSP preconditioned resid norm 1.432433111230e+04 true resid norm 7.751097283288e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.432433111230e+04 true resid norm 7.751097283288e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.678629111897e-09 true resid norm 2.873408800557e-10 ||Ae||/||Ax|| 3.707099389337e-13 1 KSP preconditioned resid norm 2.678629111897e-09 true resid norm 2.873408800557e-10 ||Ae||/||Ax|| 3.707099389337e-13 0 KSP preconditioned resid norm 2.179340507789e+04 true resid norm 3.160361945851e+06 ||Ae||/||Ax|| 3.160361945851e+06 95 KSP preconditioned resid norm 5.781488580957e-01 true resid norm 5.781488580957e-01 ||Ae||/||Ax|| 8.701849779280e-01 0 KSP preconditioned resid norm 2.306204794839e+01 true resid norm 1.753817523071e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.306204794839e+01 true resid norm 1.753817523071e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.909464122181e-12 true resid norm 1.012358671775e-12 ||Ae||/||Ax|| 5.772314727487e-13 1 KSP preconditioned resid norm 3.909464122181e-12 true resid norm 1.012358671775e-12 ||Ae||/||Ax|| 5.772314727487e-13 0 KSP preconditioned resid norm 3.566359604330e+01 true resid norm 1.033237546789e+04 ||Ae||/||Ax|| 1.033237546789e+04 1 KSP preconditioned resid norm 3.961363384402e+00 true resid norm 4.523755966710e+02 ||Ae||/||Ax|| 4.523755966710e+02 0 KSP preconditioned resid norm 1.569777113244e+04 true resid norm 5.646829084848e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.569777113244e+04 true resid norm 5.646829084848e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.960576905791e-09 true resid norm 1.003138802914e-09 ||Ae||/||Ax|| 1.776463901849e-12 1 KSP preconditioned resid norm 1.960576905791e-09 true resid norm 1.003138802914e-09 ||Ae||/||Ax|| 1.776463901849e-12 0 KSP preconditioned resid norm 2.328272999469e+04 true resid norm 1.019146903873e+07 ||Ae||/||Ax|| 1.019146903873e+07 96 KSP preconditioned resid norm 5.780939353474e-01 true resid norm 5.780939353474e-01 ||Ae||/||Ax|| 8.701023124521e-01 0 KSP preconditioned resid norm 2.523966994426e+01 true resid norm 1.427142417818e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.523966994426e+01 true resid norm 1.427142417818e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.696151373685e-12 true resid norm 2.899424767120e-12 ||Ae||/||Ax|| 2.031629591356e-12 1 KSP preconditioned resid norm 4.696151373685e-12 true resid norm 2.899424767120e-12 ||Ae||/||Ax|| 2.031629591356e-12 0 KSP preconditioned resid norm 3.822047061207e+01 true resid norm 2.877773199279e+04 ||Ae||/||Ax|| 2.877773199279e+04 0 KSP preconditioned resid norm 5.239827145472e+05 true resid norm 3.960542084533e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 5.239827145472e+05 true resid norm 3.960542084533e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.206943437103e-07 true resid norm 8.344477043618e-08 ||Ae||/||Ax|| 2.106902758641e-12 1 KSP preconditioned resid norm 1.206943437103e-07 true resid norm 8.344477043618e-08 ||Ae||/||Ax|| 2.106902758641e-12 0 KSP preconditioned resid norm 8.091905864722e+05 true resid norm 8.786451350027e+08 ||Ae||/||Ax|| 8.786451350027e+08 97 KSP preconditioned resid norm 5.780693193956e-01 true resid norm 5.780693193956e-01 ||Ae||/||Ax|| 8.700652624239e-01 0 KSP preconditioned resid norm 1.273735521175e+02 true resid norm 1.417872493295e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.273735521175e+02 true resid norm 1.417872493295e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.225895266321e-11 true resid norm 1.949742713331e-12 ||Ae||/||Ax|| 1.375118512102e-12 1 KSP preconditioned resid norm 1.225895266321e-11 true resid norm 1.949742713331e-12 ||Ae||/||Ax|| 1.375118512102e-12 0 KSP preconditioned resid norm 1.869831058212e+02 true resid norm 1.960134092692e+04 ||Ae||/||Ax|| 1.960134092692e+04 0 KSP preconditioned resid norm 2.467970681993e+06 true resid norm 2.715215672456e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.467970681993e+06 true resid norm 2.715215672456e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.219361812540e-07 true resid norm 4.024137180932e-08 ||Ae||/||Ax|| 1.482069075306e-12 1 KSP preconditioned resid norm 2.219361812540e-07 true resid norm 4.024137180932e-08 ||Ae||/||Ax|| 1.482069075306e-12 0 KSP preconditioned resid norm 3.622922508749e+06 true resid norm 4.123732289482e+08 ||Ae||/||Ax|| 4.123732289482e+08 98 KSP preconditioned resid norm 5.778578187423e-01 true resid norm 5.778578187423e-01 ||Ae||/||Ax|| 8.697469279868e-01 0 KSP preconditioned resid norm 2.420335031712e+01 true resid norm 1.801507397798e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.420335031712e+01 true resid norm 1.801507397798e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.176865923608e-12 true resid norm 1.508027966661e-12 ||Ae||/||Ax|| 8.370922975417e-13 1 KSP preconditioned resid norm 4.176865923608e-12 true resid norm 1.508027966661e-12 ||Ae||/||Ax|| 8.370922975417e-13 0 KSP preconditioned resid norm 3.645152326544e+01 true resid norm 1.328932384933e+04 ||Ae||/||Ax|| 1.328932384933e+04 1 KSP preconditioned resid norm 5.097331049424e+00 true resid norm 8.083826688367e+02 ||Ae||/||Ax|| 8.083826688367e+02 0 KSP preconditioned resid norm 2.671749989304e+04 true resid norm 1.002642116539e+03 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.671749989304e+04 true resid norm 1.002642116539e+03 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.213856477170e-09 true resid norm 2.347280397678e-09 ||Ae||/||Ax|| 2.341094951987e-12 1 KSP preconditioned resid norm 5.213856477170e-09 true resid norm 2.347280397678e-09 ||Ae||/||Ax|| 2.341094951987e-12 0 KSP preconditioned resid norm 3.996800141628e+04 true resid norm 2.015710432510e+07 ||Ae||/||Ax|| 2.015710432510e+07 99 KSP preconditioned resid norm 5.778052005289e-01 true resid norm 5.778052005289e-01 ||Ae||/||Ax|| 8.696677311186e-01 0 KSP preconditioned resid norm 2.117180191019e+01 true resid norm 1.467579483375e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.117180191019e+01 true resid norm 1.467579483375e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 6.487968255464e-12 true resid norm 3.171231925791e-12 ||Ae||/||Ax|| 2.160858721258e-12 1 KSP preconditioned resid norm 6.487968255464e-12 true resid norm 3.171231925791e-12 ||Ae||/||Ax|| 2.160858721258e-12 0 KSP preconditioned resid norm 3.225029306820e+01 true resid norm 2.814631621777e+04 ||Ae||/||Ax|| 2.814631621777e+04 0 KSP preconditioned resid norm 4.523530320889e+05 true resid norm 3.875023149071e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.523530320889e+05 true resid norm 3.875023149071e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.810687583045e-07 true resid norm 9.909937010344e-08 ||Ae||/||Ax|| 2.557387821727e-12 1 KSP preconditioned resid norm 1.810687583045e-07 true resid norm 9.909937010344e-08 ||Ae||/||Ax|| 2.557387821727e-12 0 KSP preconditioned resid norm 7.127529316003e+05 true resid norm 8.602590991178e+08 ||Ae||/||Ax|| 8.602590991178e+08 100 KSP preconditioned resid norm 5.775826742339e-01 true resid norm 5.775826742339e-01 ||Ae||/||Ax|| 8.693328017378e-01 0 KSP preconditioned resid norm 4.173866730709e+01 true resid norm 1.656068350932e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.173866730709e+01 true resid norm 1.656068350932e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.102909071988e-12 true resid norm 1.371828994781e-12 ||Ae||/||Ax|| 8.283649609085e-13 1 KSP preconditioned resid norm 5.102909071988e-12 true resid norm 1.371828994781e-12 ||Ae||/||Ax|| 8.283649609085e-13 0 KSP preconditioned resid norm 6.233517788619e+01 true resid norm 1.224436983729e+04 ||Ae||/||Ax|| 1.224436983729e+04 0 KSP preconditioned resid norm 8.081153626986e+05 true resid norm 1.727926041098e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.081153626986e+05 true resid norm 1.727926041098e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.254789580939e-07 true resid norm 3.634672700734e-08 ||Ae||/||Ax|| 2.103488583589e-12 1 KSP preconditioned resid norm 1.254789580939e-07 true resid norm 3.634672700734e-08 ||Ae||/||Ax|| 2.103488583589e-12 0 KSP preconditioned resid norm 1.205026700003e+06 true resid norm 3.215246992350e+08 ||Ae||/||Ax|| 3.215246992350e+08 101 KSP preconditioned resid norm 5.775307980729e-01 true resid norm 5.775307980729e-01 ||Ae||/||Ax|| 8.692547217496e-01 0 KSP preconditioned resid norm 3.435170014284e+01 true resid norm 1.438517571536e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.435170014284e+01 true resid norm 1.438517571536e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.260722810963e-12 true resid norm 3.112827248484e-12 ||Ae||/||Ax|| 2.163913260482e-12 1 KSP preconditioned resid norm 7.260722810963e-12 true resid norm 3.112827248484e-12 ||Ae||/||Ax|| 2.163913260482e-12 0 KSP preconditioned resid norm 5.163926752198e+01 true resid norm 2.759499368127e+04 ||Ae||/||Ax|| 2.759499368127e+04 0 KSP preconditioned resid norm 6.919550922855e+05 true resid norm 3.800160663045e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 6.919550922855e+05 true resid norm 3.800160663045e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.957221831070e-07 true resid norm 9.319415383300e-08 ||Ae||/||Ax|| 2.452374046689e-12 1 KSP preconditioned resid norm 1.957221831070e-07 true resid norm 9.319415383300e-08 ||Ae||/||Ax|| 2.452374046689e-12 0 KSP preconditioned resid norm 1.055717278577e+06 true resid norm 8.370811507176e+08 ||Ae||/||Ax|| 8.370811507176e+08 102 KSP preconditioned resid norm 5.774689532168e-01 true resid norm 5.774689532168e-01 ||Ae||/||Ax|| 8.691616376521e-01 0 KSP preconditioned resid norm 1.222577866385e+02 true resid norm 1.420473159311e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.222577866385e+02 true resid norm 1.420473159311e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.344428508248e-11 true resid norm 2.158060760638e-12 ||Ae||/||Ax|| 1.519254866938e-12 1 KSP preconditioned resid norm 1.344428508248e-11 true resid norm 2.158060760638e-12 ||Ae||/||Ax|| 1.519254866938e-12 0 KSP preconditioned resid norm 1.817007589865e+02 true resid norm 2.058936356161e+04 ||Ae||/||Ax|| 2.058936356161e+04 0 KSP preconditioned resid norm 2.369595886156e+06 true resid norm 2.848701428949e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.369595886156e+06 true resid norm 2.848701428949e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.649654983221e-07 true resid norm 5.187838733359e-08 ||Ae||/||Ax|| 1.821124067493e-12 1 KSP preconditioned resid norm 2.649654983221e-07 true resid norm 5.187838733359e-08 ||Ae||/||Ax|| 1.821124067493e-12 0 KSP preconditioned resid norm 3.523006001226e+06 true resid norm 4.718192712223e+08 ||Ae||/||Ax|| 4.718192712223e+08 103 KSP preconditioned resid norm 5.763526267619e-01 true resid norm 5.763526267619e-01 ||Ae||/||Ax|| 8.674814293494e-01 0 KSP preconditioned resid norm 1.697136665891e+01 true resid norm 1.955324008574e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.697136665891e+01 true resid norm 1.955324008574e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.305659692046e-12 true resid norm 5.152032928709e-13 ||Ae||/||Ax|| 2.634874274604e-13 1 KSP preconditioned resid norm 2.305659692046e-12 true resid norm 5.152032928709e-13 ||Ae||/||Ax|| 2.634874274604e-13 0 KSP preconditioned resid norm 2.585789437966e+01 true resid norm 4.260004083289e+03 ||Ae||/||Ax|| 4.260004083289e+03 1 KSP preconditioned resid norm 3.042227338085e+00 true resid norm 2.437924699318e+02 ||Ae||/||Ax|| 2.437924699318e+02 0 KSP preconditioned resid norm 1.786172665561e+04 true resid norm 3.672203942080e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.786172665561e+04 true resid norm 3.672203942080e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.117674500496e-09 true resid norm 2.927523198325e-10 ||Ae||/||Ax|| 7.972114960115e-13 1 KSP preconditioned resid norm 2.117674500496e-09 true resid norm 2.927523198325e-10 ||Ae||/||Ax|| 7.972114960115e-13 0 KSP preconditioned resid norm 2.668032793746e+04 true resid norm 2.836190692441e+06 ||Ae||/||Ax|| 2.836190692441e+06 104 KSP preconditioned resid norm 5.751140756815e-01 true resid norm 5.751140756815e-01 ||Ae||/||Ax|| 8.656172579868e-01 0 KSP preconditioned resid norm 4.178662399122e+01 true resid norm 1.904158424088e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 4.178662399122e+01 true resid norm 1.904158424088e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.352669269920e-12 true resid norm 9.152708855467e-13 ||Ae||/||Ax|| 4.806695041591e-13 1 KSP preconditioned resid norm 4.352669269920e-12 true resid norm 9.152708855467e-13 ||Ae||/||Ax|| 4.806695041591e-13 0 KSP preconditioned resid norm 6.271220970509e+01 true resid norm 7.352924591664e+03 ||Ae||/||Ax|| 7.352924591664e+03 0 KSP preconditioned resid norm 7.968922992108e+05 true resid norm 1.065682553344e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.968922992108e+05 true resid norm 1.065682553344e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.404499849571e-08 true resid norm 1.479317097437e-08 ||Ae||/||Ax|| 1.388140485921e-12 1 KSP preconditioned resid norm 8.404499849571e-08 true resid norm 1.479317097437e-08 ||Ae||/||Ax|| 1.388140485921e-12 0 KSP preconditioned resid norm 1.183939233126e+06 true resid norm 1.477007546814e+08 ||Ae||/||Ax|| 1.477007546814e+08 105 KSP preconditioned resid norm 5.749561994761e-01 true resid norm 5.749561994761e-01 ||Ae||/||Ax|| 8.653796349242e-01 0 KSP preconditioned resid norm 9.629549323088e+01 true resid norm 1.451840154351e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 9.629549323088e+01 true resid norm 1.451840154351e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.074371741787e-11 true resid norm 2.619883717379e-12 ||Ae||/||Ax|| 1.804526283095e-12 1 KSP preconditioned resid norm 1.074371741787e-11 true resid norm 2.619883717379e-12 ||Ae||/||Ax|| 1.804526283095e-12 0 KSP preconditioned resid norm 1.429556151408e+02 true resid norm 2.380182751776e+04 ||Ae||/||Ax|| 2.380182751776e+04 0 KSP preconditioned resid norm 1.870052902184e+06 true resid norm 3.283587561260e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.870052902184e+06 true resid norm 3.283587561260e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.042873448645e-07 true resid norm 7.432012031120e-08 ||Ae||/||Ax|| 2.263381710542e-12 1 KSP preconditioned resid norm 2.042873448645e-07 true resid norm 7.432012031120e-08 ||Ae||/||Ax|| 2.263381710542e-12 0 KSP preconditioned resid norm 2.780763673784e+06 true resid norm 6.511750937577e+08 ||Ae||/||Ax|| 6.511750937577e+08 106 KSP preconditioned resid norm 5.748966047003e-01 true resid norm 5.748966047003e-01 ||Ae||/||Ax|| 8.652899374735e-01 0 KSP preconditioned resid norm 8.306424730183e+01 true resid norm 1.405049149366e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 8.306424730183e+01 true resid norm 1.405049149366e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.962670663381e-12 true resid norm 2.842302835209e-12 ||Ae||/||Ax|| 2.022920576474e-12 1 KSP preconditioned resid norm 7.962670663381e-12 true resid norm 2.842302835209e-12 ||Ae||/||Ax|| 2.022920576474e-12 0 KSP preconditioned resid norm 1.234427131551e+02 true resid norm 2.643728907119e+04 ||Ae||/||Ax|| 2.643728907119e+04 0 KSP preconditioned resid norm 1.617784243922e+06 true resid norm 3.642368999034e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.617784243922e+06 true resid norm 3.642368999034e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.864935874265e-07 true resid norm 8.452353792722e-08 ||Ae||/||Ax|| 2.320564938633e-12 1 KSP preconditioned resid norm 1.864935874265e-07 true resid norm 8.452353792722e-08 ||Ae||/||Ax|| 2.320564938633e-12 0 KSP preconditioned resid norm 2.410992663918e+06 true resid norm 7.598164966855e+08 ||Ae||/||Ax|| 7.598164966855e+08 107 KSP preconditioned resid norm 5.742602293381e-01 true resid norm 5.742602293380e-01 ||Ae||/||Ax|| 8.643321144617e-01 0 KSP preconditioned resid norm 1.676038567028e+01 true resid norm 1.791468385419e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.676038567028e+01 true resid norm 1.791468385419e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.962647878255e-12 true resid norm 6.853297459495e-13 ||Ae||/||Ax|| 3.825519621375e-13 1 KSP preconditioned resid norm 1.962647878255e-12 true resid norm 6.853297459495e-13 ||Ae||/||Ax|| 3.825519621375e-13 0 KSP preconditioned resid norm 2.532885709747e+01 true resid norm 5.454466483346e+03 ||Ae||/||Ax|| 5.454466483346e+03 1 KSP preconditioned resid norm 3.229688487480e+00 true resid norm 3.267906245437e+02 ||Ae||/||Ax|| 3.267906245437e+02 0 KSP preconditioned resid norm 1.843164949732e+04 true resid norm 4.218893826926e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.843164949732e+04 true resid norm 4.218893826926e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.468796375580e-09 true resid norm 4.208772922021e-10 ||Ae||/||Ax|| 9.976010524747e-13 1 KSP preconditioned resid norm 1.468796375580e-09 true resid norm 4.208772922021e-10 ||Ae||/||Ax|| 9.976010524747e-13 0 KSP preconditioned resid norm 2.745803820367e+04 true resid norm 3.939322544201e+06 ||Ae||/||Ax|| 3.939322544201e+06 108 KSP preconditioned resid norm 5.739811873555e-01 true resid norm 5.739811873555e-01 ||Ae||/||Ax|| 8.639121220359e-01 0 KSP preconditioned resid norm 3.808636715213e+01 true resid norm 1.781146119626e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.808636715213e+01 true resid norm 1.781146119626e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.250981783333e-12 true resid norm 9.707995399207e-13 ||Ae||/||Ax|| 5.450420542278e-13 1 KSP preconditioned resid norm 5.250981783333e-12 true resid norm 9.707995399207e-13 ||Ae||/||Ax|| 5.450420542278e-13 0 KSP preconditioned resid norm 5.732830282847e+01 true resid norm 8.318751076374e+03 ||Ae||/||Ax|| 8.318751076374e+03 1 KSP preconditioned resid norm 6.350839005809e+00 true resid norm 4.337752163776e+02 ||Ae||/||Ax|| 4.337752163776e+02 0 KSP preconditioned resid norm 3.460910136920e+04 true resid norm 5.622099019164e+02 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 3.460910136920e+04 true resid norm 5.622099019164e+02 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.147780388064e-09 true resid norm 9.213377499091e-10 ||Ae||/||Ax|| 1.638778944961e-12 1 KSP preconditioned resid norm 3.147780388064e-09 true resid norm 9.213377499091e-10 ||Ae||/||Ax|| 1.638778944961e-12 0 KSP preconditioned resid norm 5.148608005798e+04 true resid norm 8.314771885136e+06 ||Ae||/||Ax|| 8.314771885136e+06 109 KSP preconditioned resid norm 5.739472011349e-01 true resid norm 5.739472011349e-01 ||Ae||/||Ax|| 8.638609686033e-01 0 KSP preconditioned resid norm 7.237935221900e+01 true resid norm 1.429408161357e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 7.237935221900e+01 true resid norm 1.429408161357e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.128641055025e-12 true resid norm 2.862624540144e-12 ||Ae||/||Ax|| 2.002664191749e-12 1 KSP preconditioned resid norm 7.128641055025e-12 true resid norm 2.862624540144e-12 ||Ae||/||Ax|| 2.002664191749e-12 0 KSP preconditioned resid norm 1.075650813158e+02 true resid norm 2.636274829393e+04 ||Ae||/||Ax|| 2.636274829393e+04 0 KSP preconditioned resid norm 1.411126440713e+06 true resid norm 3.632049357939e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.411126440713e+06 true resid norm 3.632049357939e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.769173357385e-07 true resid norm 8.417464772249e-08 ||Ae||/||Ax|| 2.317552418127e-12 1 KSP preconditioned resid norm 1.769173357385e-07 true resid norm 8.417464772249e-08 ||Ae||/||Ax|| 2.317552418127e-12 0 KSP preconditioned resid norm 2.101840499421e+06 true resid norm 7.697095423823e+08 ||Ae||/||Ax|| 7.697095423823e+08 110 KSP preconditioned resid norm 5.739335924829e-01 true resid norm 5.739335924829e-01 ||Ae||/||Ax|| 8.638404859121e-01 0 KSP preconditioned resid norm 1.062282481225e+02 true resid norm 1.402026143656e+00 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 1.062282481225e+02 true resid norm 1.402026143656e+00 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 8.851539111518e-12 true resid norm 2.527921035453e-12 ||Ae||/||Ax|| 1.803048428799e-12 1 KSP preconditioned resid norm 8.851539111518e-12 true resid norm 2.527921035453e-12 ||Ae||/||Ax|| 1.803048428799e-12 0 KSP preconditioned resid norm 1.575377524634e+02 true resid norm 2.367845612632e+04 ||Ae||/||Ax|| 2.367845612632e+04 0 KSP preconditioned resid norm 2.061767162809e+06 true resid norm 3.267618142454e+04 ||Ae||/||Ax|| 1.000000000000e+00 0 KSP preconditioned resid norm 2.061767162809e+06 true resid norm 3.267618142454e+04 ||Ae||/||Ax|| 1.000000000000e+00 1 KSP preconditioned resid norm 1.860389500586e-07 true resid norm 6.799701708474e-08 ||Ae||/||Ax|| 2.080935229282e-12 1 KSP preconditioned resid norm 1.860389500586e-07 true resid norm 6.799701708474e-08 ||Ae||/||Ax|| 2.080935229282e-12 0 KSP preconditioned resid norm 3.058445347287e+06 true resid norm 6.285663338975e+08 ||Ae||/||Ax|| 6.285663338975e+08 Thanks again, Hal Barry Smith wrote: > > No, I want it on the nonworking two-level case. Isn't that your goal > to do multigrid? If one level is working and you are happy with it then > why send the email? > > Barry > > On Aug 26, 2009, at 3:45 PM, Hal Finkel wrote: > >> Barry, >> >> Here is the output with those options in the (working) 1-level case on >> 1 cpu. Is that what you wanted? >> >> 0 SNES Function norm 6.643976542462e-01 >> 0 KSP preconditioned resid norm 6.643976542462e-01 true resid norm >> 6.643976542462e-01 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.438911631652e-03 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.438911631652e-03 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.141505633982e-04 true resid >> norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 >> 1 KSP preconditioned resid norm 5.141505633982e-04 true resid >> norm 1.970012522193e-01 ||Ae||/||Ax|| 1.970012522193e-01 >> 1 KSP preconditioned resid norm 4.790430472989e-02 true resid norm >> 4.790430472989e-02 ||Ae||/||Ax|| 7.210185710881e-02 >> 0 KSP preconditioned resid norm 8.314136308720e-03 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.314136308720e-03 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.510029589596e-03 true resid >> norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 >> 1 KSP preconditioned resid norm 4.510029589596e-03 true resid >> norm 5.934134865465e-01 ||Ae||/||Ax|| 5.934134865465e-01 >> 2 KSP preconditioned resid norm 8.487458086567e-03 true resid norm >> 8.487458086567e-03 ||Ae||/||Ax|| 1.277466594339e-02 >> 0 KSP preconditioned resid norm 6.119493660032e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 6.119493660032e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.236002849750e-02 true resid >> norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 >> 1 KSP preconditioned resid norm 2.236002849750e-02 true resid >> norm 7.814908146189e-01 ||Ae||/||Ax|| 7.814908146189e-01 >> 3 KSP preconditioned resid norm 5.965466932886e-03 true resid norm >> 5.965466932886e-03 ||Ae||/||Ax|| 8.978759775506e-03 >> 0 KSP preconditioned resid norm 1.765708232662e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.765708232662e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.422901024605e-02 true resid >> norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 >> 1 KSP preconditioned resid norm 2.422901024605e-02 true resid >> norm 6.919985015683e-01 ||Ae||/||Ax|| 6.919985015683e-01 >> 4 KSP preconditioned resid norm 4.622977909431e-03 true resid norm >> 4.622977909431e-03 ||Ae||/||Ax|| 6.958149054087e-03 >> 0 KSP preconditioned resid norm 8.428635776507e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.428635776507e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.071459298111e-02 true resid >> norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 >> 1 KSP preconditioned resid norm 1.071459298111e-02 true resid >> norm 4.423943234439e-01 ||Ae||/||Ax|| 4.423943234439e-01 >> 5 KSP preconditioned resid norm 2.949006884559e-03 true resid norm >> 2.949006884560e-03 ||Ae||/||Ax|| 4.438617243322e-03 >> 0 KSP preconditioned resid norm 4.221520918573e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.221520918573e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.795668593850e-02 true resid >> norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 >> 1 KSP preconditioned resid norm 1.795668593850e-02 true resid >> norm 8.064343336541e-01 ||Ae||/||Ax|| 8.064343336541e-01 >> 6 KSP preconditioned resid norm 1.327402359192e-03 true resid norm >> 1.327402359192e-03 ||Ae||/||Ax|| 1.997903440370e-03 >> 0 KSP preconditioned resid norm 3.239330680409e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.239330680409e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.103583195381e-02 true resid >> norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 >> 1 KSP preconditioned resid norm 1.103583195381e-02 true resid >> norm 5.910503837805e-01 ||Ae||/||Ax|| 5.910503837805e-01 >> 7 KSP preconditioned resid norm 5.579319891254e-04 true resid norm >> 5.579319891255e-04 ||Ae||/||Ax|| 8.397561092512e-04 >> 0 KSP preconditioned resid norm 2.166256693843e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.166256693843e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.527142494434e-02 true resid >> norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 >> 1 KSP preconditioned resid norm 2.527142494434e-02 true resid >> norm 3.938455942305e-01 ||Ae||/||Ax|| 3.938455942305e-01 >> 8 KSP preconditioned resid norm 1.841437265702e-04 true resid norm >> 1.841437265703e-04 ||Ae||/||Ax|| 2.771589053535e-04 >> 0 KSP preconditioned resid norm 1.097958713832e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.097958713832e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.070826208815e-02 true resid >> norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 >> 1 KSP preconditioned resid norm 4.070826208815e-02 true resid >> norm 5.960016421595e-01 ||Ae||/||Ax|| 5.960016421595e-01 >> 9 KSP preconditioned resid norm 6.545738966234e-05 true resid norm >> 6.545738966232e-05 ||Ae||/||Ax|| 9.852140392726e-05 >> 0 KSP preconditioned resid norm 9.668719808206e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.668719808206e-02 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 7.944112488019e-02 true resid >> norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 >> 1 KSP preconditioned resid norm 7.944112488019e-02 true resid >> norm 7.461779305230e-01 ||Ae||/||Ax|| 7.461779305230e-01 >> 10 KSP preconditioned resid norm 4.125461342831e-05 true resid norm >> 4.125461342821e-05 ||Ae||/||Ax|| 6.209325569492e-05 >> 0 KSP preconditioned resid norm 3.008151203349e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.008151203349e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 7.247738676584e-02 true resid >> norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 >> 1 KSP preconditioned resid norm 7.247738676584e-02 true resid >> norm 7.832960673267e-01 ||Ae||/||Ax|| 7.832960673267e-01 >> 11 KSP preconditioned resid norm 3.280778128691e-05 true resid norm >> 3.280778128689e-05 ||Ae||/||Ax|| 4.937973678446e-05 >> 0 KSP preconditioned resid norm 9.091470733399e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.091470733399e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.215403990109e-01 true resid >> norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 >> 1 KSP preconditioned resid norm 1.215403990109e-01 true resid >> norm 7.059207029764e-01 ||Ae||/||Ax|| 7.059207029764e-01 >> 12 KSP preconditioned resid norm 2.968550831459e-05 true resid norm >> 2.968550831435e-05 ||Ae||/||Ax|| 4.468033281669e-05 >> 0 KSP preconditioned resid norm 1.024246289230e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.024246289230e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 9.302673603327e-02 true resid >> norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 >> 1 KSP preconditioned resid norm 9.302673603327e-02 true resid >> norm 5.985130004903e-01 ||Ae||/||Ax|| 5.985130004903e-01 >> 13 KSP preconditioned resid norm 2.521514028222e-05 true resid norm >> 2.521514028196e-05 ||Ae||/||Ax|| 3.795188035479e-05 >> 0 KSP preconditioned resid norm 5.990912160630e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.990912160630e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 9.781990197263e-02 true resid >> norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 >> 1 KSP preconditioned resid norm 9.781990197263e-02 true resid >> norm 5.480521896855e-01 ||Ae||/||Ax|| 5.480521896855e-01 >> 14 KSP preconditioned resid norm 1.978375072845e-05 true resid norm >> 1.978375072804e-05 ||Ae||/||Ax|| 2.977697257298e-05 >> 0 KSP preconditioned resid norm 1.008331187053e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.008331187053e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.294954965258e-01 true resid >> norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 >> 1 KSP preconditioned resid norm 2.294954965258e-01 true resid >> norm 6.339103426898e-01 ||Ae||/||Ax|| 6.339103426898e-01 >> 15 KSP preconditioned resid norm 1.488645974384e-05 true resid norm >> 1.488645974344e-05 ||Ae||/||Ax|| 2.240594867893e-05 >> 0 KSP preconditioned resid norm 1.849227047229e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.849227047229e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.919984907088e-01 true resid >> norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 >> 1 KSP preconditioned resid norm 2.919984907088e-01 true resid >> norm 5.183188121423e-01 ||Ae||/||Ax|| 5.183188121423e-01 >> 16 KSP preconditioned resid norm 7.094849419546e-06 true resid norm >> 7.094849419588e-06 ||Ae||/||Ax|| 1.067861900813e-05 >> 0 KSP preconditioned resid norm 1.388766130430e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.388766130430e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.878119571370e-01 true resid >> norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 >> 1 KSP preconditioned resid norm 3.878119571370e-01 true resid >> norm 7.032718519224e-01 ||Ae||/||Ax|| 7.032718519224e-01 >> 17 KSP preconditioned resid norm 4.097305136804e-06 true resid norm >> 4.097305136552e-06 ||Ae||/||Ax|| 6.166947023919e-06 >> 1 SNES Function norm 5.850920566283e-06 >> 0 KSP preconditioned resid norm 5.850920566283e-06 true resid norm >> 5.850920566283e-06 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.276021960717e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.276021960717e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.809774037292e-01 true resid >> norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 >> 1 KSP preconditioned resid norm 3.809774037292e-01 true resid >> norm 7.868108837109e-01 ||Ae||/||Ax|| 7.868108837109e-01 >> 1 KSP preconditioned resid norm 3.947856942305e-06 true resid norm >> 3.947856942305e-06 ||Ae||/||Ax|| 6.747411621096e-01 >> 0 KSP preconditioned resid norm 2.497905632896e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.497905632896e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.608088417629e-01 true resid >> norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 >> 1 KSP preconditioned resid norm 5.608088417629e-01 true resid >> norm 8.391706412367e-01 ||Ae||/||Ax|| 8.391706412367e-01 >> 2 KSP preconditioned resid norm 3.305771768709e-06 true resid norm >> 3.305771768709e-06 ||Ae||/||Ax|| 5.650002817948e-01 >> 0 KSP preconditioned resid norm 7.301425383804e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.301425383804e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.476223530528e-01 true resid >> norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 >> 1 KSP preconditioned resid norm 3.476223530528e-01 true resid >> norm 5.563009832684e-01 ||Ae||/||Ax|| 5.563009832684e-01 >> 3 KSP preconditioned resid norm 3.007329187251e-06 true resid norm >> 3.007329187251e-06 ||Ae||/||Ax|| 5.139924825815e-01 >> 0 KSP preconditioned resid norm 2.880397337024e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.880397337024e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.471977831965e-01 true resid >> norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 >> 1 KSP preconditioned resid norm 4.471977831965e-01 true resid >> norm 7.798621978217e-01 ||Ae||/||Ax|| 7.798621978217e-01 >> 4 KSP preconditioned resid norm 3.001204159469e-06 true resid norm >> 3.001204159469e-06 ||Ae||/||Ax|| 5.129456340194e-01 >> 0 KSP preconditioned resid norm 4.619989941290e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.619989941290e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.243648576810e-01 true resid >> norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 >> 1 KSP preconditioned resid norm 3.243648576810e-01 true resid >> norm 1.108094778155e+00 ||Ae||/||Ax|| 1.108094778155e+00 >> 5 KSP preconditioned resid norm 3.001115709599e-06 true resid norm >> 3.001115709599e-06 ||Ae||/||Ax|| 5.129305167624e-01 >> 0 KSP preconditioned resid norm 4.829994865746e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.829994865746e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.325734548745e-01 true resid >> norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 >> 1 KSP preconditioned resid norm 4.325734548745e-01 true resid >> norm 1.088867367033e+00 ||Ae||/||Ax|| 1.088867367033e+00 >> 6 KSP preconditioned resid norm 2.974277803753e-06 true resid norm >> 2.974277803753e-06 ||Ae||/||Ax|| 5.083435623606e-01 >> 0 KSP preconditioned resid norm 1.381129102428e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.381129102428e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.098881160663e-01 true resid >> norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 >> 1 KSP preconditioned resid norm 3.098881160663e-01 true resid >> norm 6.431427158439e-01 ||Ae||/||Ax|| 6.431427158439e-01 >> 7 KSP preconditioned resid norm 2.816838388951e-06 true resid norm >> 2.816838388951e-06 ||Ae||/||Ax|| 4.814350762483e-01 >> 0 KSP preconditioned resid norm 1.559573417426e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.559573417426e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.532723445272e-01 true resid >> norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 >> 1 KSP preconditioned resid norm 2.532723445272e-01 true resid >> norm 8.138557633078e-01 ||Ae||/||Ax|| 8.138557633078e-01 >> 8 KSP preconditioned resid norm 2.572324455447e-06 true resid norm >> 2.572324455447e-06 ||Ae||/||Ax|| 4.396443989123e-01 >> 0 KSP preconditioned resid norm 5.946831260059e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.946831260059e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.929968207615e-01 true resid >> norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 >> 1 KSP preconditioned resid norm 3.929968207615e-01 true resid >> norm 1.354756640752e+00 ||Ae||/||Ax|| 1.354756640752e+00 >> 9 KSP preconditioned resid norm 2.177455315353e-06 true resid norm >> 2.177455315353e-06 ||Ae||/||Ax|| 3.721560206954e-01 >> 0 KSP preconditioned resid norm 5.315723299063e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.315723299063e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.628504511585e-01 true resid >> norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 >> 1 KSP preconditioned resid norm 3.628504511585e-01 true resid >> norm 1.244733734536e+00 ||Ae||/||Ax|| 1.244733734536e+00 >> 10 KSP preconditioned resid norm 1.409067985559e-06 true resid norm >> 1.409067985559e-06 ||Ae||/||Ax|| 2.408284251334e-01 >> 0 KSP preconditioned resid norm 1.522258580690e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.522258580690e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.414238177111e-01 true resid >> norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 >> 1 KSP preconditioned resid norm 2.414238177111e-01 true resid >> norm 8.091416034904e-01 ||Ae||/||Ax|| 8.091416034904e-01 >> 11 KSP preconditioned resid norm 1.001503517728e-06 true resid norm >> 1.001503517728e-06 ||Ae||/||Ax|| 1.711702468667e-01 >> 0 KSP preconditioned resid norm 8.244553153983e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.244553153983e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.244218748685e-01 true resid >> norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 >> 1 KSP preconditioned resid norm 2.244218748685e-01 true resid >> norm 6.292257324755e-01 ||Ae||/||Ax|| 6.292257324755e-01 >> 12 KSP preconditioned resid norm 7.556215980081e-07 true resid norm >> 7.556215980082e-07 ||Ae||/||Ax|| 1.291457625254e-01 >> 0 KSP preconditioned resid norm 1.857974243745e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.857974243745e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.451523414794e-01 true resid >> norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 >> 1 KSP preconditioned resid norm 3.451523414794e-01 true resid >> norm 6.449941313821e-01 ||Ae||/||Ax|| 6.449941313821e-01 >> 13 KSP preconditioned resid norm 5.872140840954e-07 true resid norm >> 5.872140840955e-07 ||Ae||/||Ax|| 1.003626826656e-01 >> 0 KSP preconditioned resid norm 2.611648570586e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.611648570586e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.740043981340e-01 true resid >> norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 >> 1 KSP preconditioned resid norm 2.740043981340e-01 true resid >> norm 6.369784579675e-01 ||Ae||/||Ax|| 6.369784579675e-01 >> 14 KSP preconditioned resid norm 4.352350316840e-07 true resid norm >> 4.352350316841e-07 ||Ae||/||Ax|| 7.438744497613e-02 >> 0 KSP preconditioned resid norm 2.003016340281e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.003016340281e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.073409352245e-01 true resid >> norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 >> 1 KSP preconditioned resid norm 4.073409352245e-01 true resid >> norm 6.562979155542e-01 ||Ae||/||Ax|| 6.562979155542e-01 >> 15 KSP preconditioned resid norm 3.052867274622e-07 true resid norm >> 3.052867274624e-07 ||Ae||/||Ax|| 5.217755462648e-02 >> 0 KSP preconditioned resid norm 1.286639096168e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.286639096168e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.656621503377e-01 true resid >> norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 >> 1 KSP preconditioned resid norm 3.656621503377e-01 true resid >> norm 6.308753365212e-01 ||Ae||/||Ax|| 6.308753365212e-01 >> 16 KSP preconditioned resid norm 1.955997188783e-07 true resid norm >> 1.955997188782e-07 ||Ae||/||Ax|| 3.343058868468e-02 >> 0 KSP preconditioned resid norm 1.335903581256e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.335903581256e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.001895431683e-01 true resid >> norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 >> 1 KSP preconditioned resid norm 3.001895431683e-01 true resid >> norm 6.789558143016e-01 ||Ae||/||Ax|| 6.789558143016e-01 >> 17 KSP preconditioned resid norm 1.567782688997e-07 true resid norm >> 1.567782688996e-07 ||Ae||/||Ax|| 2.679548750039e-02 >> 0 KSP preconditioned resid norm 1.651696143482e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.651696143482e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.713972348795e-01 true resid >> norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 >> 1 KSP preconditioned resid norm 2.713972348795e-01 true resid >> norm 5.033537683058e-01 ||Ae||/||Ax|| 5.033537683058e-01 >> 18 KSP preconditioned resid norm 1.010718356377e-07 true resid norm >> 1.010718356378e-07 ||Ae||/||Ax|| 1.727451851256e-02 >> 0 KSP preconditioned resid norm 1.011081720758e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.011081720758e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.527570671391e-01 true resid >> norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 >> 1 KSP preconditioned resid norm 5.527570671391e-01 true resid >> norm 7.947330071383e-01 ||Ae||/||Ax|| 7.947330071383e-01 >> 19 KSP preconditioned resid norm 8.997938714701e-08 true resid norm >> 8.997938714709e-08 ||Ae||/||Ax|| 1.537867180519e-02 >> 0 KSP preconditioned resid norm 1.537108246495e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.537108246495e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.298772065070e-01 true resid >> norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 >> 1 KSP preconditioned resid norm 4.298772065070e-01 true resid >> norm 5.876338587223e-01 ||Ae||/||Ax|| 5.876338587223e-01 >> 20 KSP preconditioned resid norm 8.711914509357e-08 true resid norm >> 8.711914509361e-08 ||Ae||/||Ax|| 1.488981846646e-02 >> 0 KSP preconditioned resid norm 1.923097362850e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.923097362850e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.075948671586e-01 true resid >> norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 >> 1 KSP preconditioned resid norm 3.075948671586e-01 true resid >> norm 4.631591237704e-01 ||Ae||/||Ax|| 4.631591237704e-01 >> 21 KSP preconditioned resid norm 8.323067409578e-08 true resid norm >> 8.323067409577e-08 ||Ae||/||Ax|| 1.422522715065e-02 >> 0 KSP preconditioned resid norm 9.819276869885e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.819276869885e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.279383354509e-01 true resid >> norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 >> 1 KSP preconditioned resid norm 4.279383354509e-01 true resid >> norm 6.842834730228e-01 ||Ae||/||Ax|| 6.842834730228e-01 >> 22 KSP preconditioned resid norm 7.909034553981e-08 true resid norm >> 7.909034553913e-08 ||Ae||/||Ax|| 1.351759003445e-02 >> 0 KSP preconditioned resid norm 9.582482498733e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.582482498733e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.872542232901e-01 true resid >> norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 >> 1 KSP preconditioned resid norm 4.872542232901e-01 true resid >> norm 6.520743487433e-01 ||Ae||/||Ax|| 6.520743487433e-01 >> 23 KSP preconditioned resid norm 7.583943619887e-08 true resid norm >> 7.583943619831e-08 ||Ae||/||Ax|| 1.296196646992e-02 >> 0 KSP preconditioned resid norm 1.129055084162e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.129055084162e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.940290407027e-01 true resid >> norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 >> 1 KSP preconditioned resid norm 3.940290407027e-01 true resid >> norm 5.533623127492e-01 ||Ae||/||Ax|| 5.533623127492e-01 >> 24 KSP preconditioned resid norm 7.198974227007e-08 true resid norm >> 7.198974226927e-08 ||Ae||/||Ax|| 1.230400266996e-02 >> 0 KSP preconditioned resid norm 7.922461164088e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.922461164088e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.568631157402e-01 true resid >> norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 >> 1 KSP preconditioned resid norm 3.568631157402e-01 true resid >> norm 6.784746707201e-01 ||Ae||/||Ax|| 6.784746707201e-01 >> 25 KSP preconditioned resid norm 6.969681693595e-08 true resid norm >> 6.969681693509e-08 ||Ae||/||Ax|| 1.191211129010e-02 >> 0 KSP preconditioned resid norm 8.506106408028e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.506106408028e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.674832821714e-01 true resid >> norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 >> 1 KSP preconditioned resid norm 2.674832821714e-01 true resid >> norm 5.592992824796e-01 ||Ae||/||Ax|| 5.592992824796e-01 >> 26 KSP preconditioned resid norm 6.713093474885e-08 true resid norm >> 6.713093474791e-08 ||Ae||/||Ax|| 1.147356796036e-02 >> 0 KSP preconditioned resid norm 7.107668950850e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.107668950850e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.080542740675e-01 true resid >> norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 >> 1 KSP preconditioned resid norm 3.080542740675e-01 true resid >> norm 6.120847770454e-01 ||Ae||/||Ax|| 6.120847770454e-01 >> 27 KSP preconditioned resid norm 6.316951623357e-08 true resid norm >> 6.316951623251e-08 ||Ae||/||Ax|| 1.079650894537e-02 >> 0 KSP preconditioned resid norm 9.816834307897e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.816834307897e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.035636714372e-01 true resid >> norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 >> 1 KSP preconditioned resid norm 3.035636714372e-01 true resid >> norm 5.561308623772e-01 ||Ae||/||Ax|| 5.561308623772e-01 >> 28 KSP preconditioned resid norm 5.685161562141e-08 true resid norm >> 5.685161562024e-08 ||Ae||/||Ax|| 9.716695856008e-03 >> 0 KSP preconditioned resid norm 1.017862349675e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.017862349675e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.438638918414e-01 true resid >> norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 >> 1 KSP preconditioned resid norm 3.438638918414e-01 true resid >> norm 6.483000200066e-01 ||Ae||/||Ax|| 6.483000200066e-01 >> 29 KSP preconditioned resid norm 5.170566864610e-08 true resid norm >> 5.170566864501e-08 ||Ae||/||Ax|| 8.837185201757e-03 >> 0 KSP preconditioned resid norm 8.897697988733e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.897697988733e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.213181899588e-01 true resid >> norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 >> 1 KSP preconditioned resid norm 3.213181899588e-01 true resid >> norm 6.348471232738e-01 ||Ae||/||Ax|| 6.348471232738e-01 >> 30 KSP preconditioned resid norm 4.885061460285e-08 true resid norm >> 4.885061460285e-08 ||Ae||/||Ax|| 8.349218563034e-03 >> 0 KSP preconditioned resid norm 7.100457326024e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.100457326024e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 6.955812925302e-01 true resid >> norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 >> 1 KSP preconditioned resid norm 6.955812925302e-01 true resid >> norm 9.726932992087e-01 ||Ae||/||Ax|| 9.726932992087e-01 >> 31 KSP preconditioned resid norm 4.725291213013e-08 true resid norm >> 4.725291213007e-08 ||Ae||/||Ax|| 8.076149999775e-03 >> 0 KSP preconditioned resid norm 4.185203199010e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.185203199010e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.196403938530e-01 true resid >> norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 >> 1 KSP preconditioned resid norm 2.196403938530e-01 true resid >> norm 4.016140042970e-01 ||Ae||/||Ax|| 4.016140042970e-01 >> 32 KSP preconditioned resid norm 4.547823851359e-08 true resid norm >> 4.547823851362e-08 ||Ae||/||Ax|| 7.772834718641e-03 >> 0 KSP preconditioned resid norm 6.581109982585e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 6.581109982585e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.128181724937e-01 true resid >> norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 >> 1 KSP preconditioned resid norm 3.128181724937e-01 true resid >> norm 7.298286188824e-01 ||Ae||/||Ax|| 7.298286188824e-01 >> 33 KSP preconditioned resid norm 4.468618222477e-08 true resid norm >> 4.468618222483e-08 ||Ae||/||Ax|| 7.637461783764e-03 >> 0 KSP preconditioned resid norm 8.120341282592e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.120341282592e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.218250119940e-01 true resid >> norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 >> 1 KSP preconditioned resid norm 2.218250119940e-01 true resid >> norm 4.572201876132e-01 ||Ae||/||Ax|| 4.572201876132e-01 >> 34 KSP preconditioned resid norm 4.314354409778e-08 true resid norm >> 4.314354409782e-08 ||Ae||/||Ax|| 7.373804448217e-03 >> 0 KSP preconditioned resid norm 5.778947005677e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.778947005677e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.877938034235e-01 true resid >> norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 >> 1 KSP preconditioned resid norm 3.877938034235e-01 true resid >> norm 6.868568173397e-01 ||Ae||/||Ax|| 6.868568173397e-01 >> 35 KSP preconditioned resid norm 4.111383309670e-08 true resid norm >> 4.111383309672e-08 ||Ae||/||Ax|| 7.026899892239e-03 >> 0 KSP preconditioned resid norm 7.589832929724e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.589832929724e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.735429451288e-01 true resid >> norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 >> 1 KSP preconditioned resid norm 2.735429451288e-01 true resid >> norm 6.133540321550e-01 ||Ae||/||Ax|| 6.133540321550e-01 >> 36 KSP preconditioned resid norm 3.896413302095e-08 true resid norm >> 3.896413302095e-08 ||Ae||/||Ax|| 6.659487610460e-03 >> 0 KSP preconditioned resid norm 9.173185113175e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.173185113175e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.052187383165e-01 true resid >> norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 >> 1 KSP preconditioned resid norm 3.052187383165e-01 true resid >> norm 6.655288549919e-01 ||Ae||/||Ax|| 6.655288549919e-01 >> 37 KSP preconditioned resid norm 3.770537943429e-08 true resid norm >> 3.770537943431e-08 ||Ae||/||Ax|| 6.444349911635e-03 >> 0 KSP preconditioned resid norm 1.137492649970e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.137492649970e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.044329763751e-01 true resid >> norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 >> 1 KSP preconditioned resid norm 3.044329763751e-01 true resid >> norm 6.147370239994e-01 ||Ae||/||Ax|| 6.147370239994e-01 >> 38 KSP preconditioned resid norm 3.694716763304e-08 true resid norm >> 3.694716763306e-08 ||Ae||/||Ax|| 6.314761448988e-03 >> 0 KSP preconditioned resid norm 1.223189932062e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.223189932062e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.107514686273e-01 true resid >> norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 >> 1 KSP preconditioned resid norm 2.107514686273e-01 true resid >> norm 5.789372321326e-01 ||Ae||/||Ax|| 5.789372321326e-01 >> 39 KSP preconditioned resid norm 3.634742486594e-08 true resid norm >> 3.634742486593e-08 ||Ae||/||Ax|| 6.212257448064e-03 >> 0 KSP preconditioned resid norm 1.192363148419e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.192363148419e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.860271918718e-01 true resid >> norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 >> 1 KSP preconditioned resid norm 2.860271918718e-01 true resid >> norm 5.850260965540e-01 ||Ae||/||Ax|| 5.850260965540e-01 >> 40 KSP preconditioned resid norm 3.561782722372e-08 true resid norm >> 3.561782722374e-08 ||Ae||/||Ax|| 6.087559525075e-03 >> 0 KSP preconditioned resid norm 8.881324685300e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 8.881324685300e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.642271412468e-01 true resid >> norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 >> 1 KSP preconditioned resid norm 2.642271412468e-01 true resid >> norm 6.637278122540e-01 ||Ae||/||Ax|| 6.637278122540e-01 >> 41 KSP preconditioned resid norm 3.506823410956e-08 true resid norm >> 3.506823410956e-08 ||Ae||/||Ax|| 5.993626765614e-03 >> 0 KSP preconditioned resid norm 9.513827527161e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 9.513827527161e-01 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.263980500382e-01 true resid >> norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 >> 1 KSP preconditioned resid norm 2.263980500382e-01 true resid >> norm 6.267230897905e-01 ||Ae||/||Ax|| 6.267230897905e-01 >> 42 KSP preconditioned resid norm 3.466236527477e-08 true resid norm >> 3.466236527477e-08 ||Ae||/||Ax|| 5.924258393546e-03 >> 0 KSP preconditioned resid norm 1.407625976444e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.407625976444e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.841824427477e-01 true resid >> norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 >> 1 KSP preconditioned resid norm 1.841824427477e-01 true resid >> norm 5.311814594204e-01 ||Ae||/||Ax|| 5.311814594204e-01 >> 43 KSP preconditioned resid norm 3.394294635279e-08 true resid norm >> 3.394294635284e-08 ||Ae||/||Ax|| 5.801300148978e-03 >> 0 KSP preconditioned resid norm 1.536263909707e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.536263909707e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.972519671057e-01 true resid >> norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 >> 1 KSP preconditioned resid norm 2.972519671057e-01 true resid >> norm 5.927201357618e-01 ||Ae||/||Ax|| 5.927201357618e-01 >> 44 KSP preconditioned resid norm 3.248296724023e-08 true resid norm >> 3.248296724025e-08 ||Ae||/||Ax|| 5.551770336352e-03 >> 0 KSP preconditioned resid norm 1.783804105384e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.783804105384e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.722881763352e-01 true resid >> norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 >> 1 KSP preconditioned resid norm 3.722881763352e-01 true resid >> norm 5.786098812309e-01 ||Ae||/||Ax|| 5.786098812309e-01 >> 45 KSP preconditioned resid norm 2.950320556254e-08 true resid norm >> 2.950320556258e-08 ||Ae||/||Ax|| 5.042489507138e-03 >> 0 KSP preconditioned resid norm 2.201496546447e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.201496546447e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.087649817784e-01 true resid >> norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 >> 1 KSP preconditioned resid norm 4.087649817784e-01 true resid >> norm 5.074174843604e-01 ||Ae||/||Ax|| 5.074174843604e-01 >> 46 KSP preconditioned resid norm 2.319584580786e-08 true resid norm >> 2.319584580791e-08 ||Ae||/||Ax|| 3.964477990280e-03 >> 0 KSP preconditioned resid norm 1.736355434315e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.736355434315e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 6.867386399797e-01 true resid >> norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 >> 1 KSP preconditioned resid norm 6.867386399797e-01 true resid >> norm 7.098722583695e-01 ||Ae||/||Ax|| 7.098722583695e-01 >> 47 KSP preconditioned resid norm 2.008430277340e-08 true resid norm >> 2.008430277343e-08 ||Ae||/||Ax|| 3.432673977693e-03 >> 0 KSP preconditioned resid norm 1.488127968937e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.488127968937e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.516786708506e-01 true resid >> norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 >> 1 KSP preconditioned resid norm 5.516786708506e-01 true resid >> norm 6.355423307226e-01 ||Ae||/||Ax|| 6.355423307226e-01 >> 48 KSP preconditioned resid norm 1.854206408497e-08 true resid norm >> 1.854206408497e-08 ||Ae||/||Ax|| 3.169084911497e-03 >> 0 KSP preconditioned resid norm 1.380328436388e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.380328436388e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.575661767087e-01 true resid >> norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 >> 1 KSP preconditioned resid norm 4.575661767087e-01 true resid >> norm 6.487066138646e-01 ||Ae||/||Ax|| 6.487066138646e-01 >> 49 KSP preconditioned resid norm 1.665547510085e-08 true resid norm >> 1.665547510085e-08 ||Ae||/||Ax|| 2.846641808270e-03 >> 0 KSP preconditioned resid norm 2.313536103290e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.313536103290e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.305138630648e-01 true resid >> norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 >> 1 KSP preconditioned resid norm 5.305138630648e-01 true resid >> norm 7.334968078908e-01 ||Ae||/||Ax|| 7.334968078908e-01 >> 50 KSP preconditioned resid norm 1.462283172240e-08 true resid norm >> 1.462283172247e-08 ||Ae||/||Ax|| 2.499236070088e-03 >> 0 KSP preconditioned resid norm 3.213123410447e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.213123410447e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.224884980518e-01 true resid >> norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 >> 1 KSP preconditioned resid norm 5.224884980518e-01 true resid >> norm 7.561756407001e-01 ||Ae||/||Ax|| 7.561756407001e-01 >> 51 KSP preconditioned resid norm 1.223657774649e-08 true resid norm >> 1.223657774655e-08 ||Ae||/||Ax|| 2.091393586347e-03 >> 0 KSP preconditioned resid norm 2.023552916685e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.023552916685e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.574715128775e-01 true resid >> norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 >> 1 KSP preconditioned resid norm 5.574715128775e-01 true resid >> norm 6.980817617824e-01 ||Ae||/||Ax|| 6.980817617824e-01 >> 52 KSP preconditioned resid norm 9.853979905043e-09 true resid norm >> 9.853979905046e-09 ||Ae||/||Ax|| 1.684175984516e-03 >> 0 KSP preconditioned resid norm 1.170745734096e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.170745734096e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 4.134603323396e-01 true resid >> norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 >> 1 KSP preconditioned resid norm 4.134603323396e-01 true resid >> norm 6.348912278346e-01 ||Ae||/||Ax|| 6.348912278346e-01 >> 53 KSP preconditioned resid norm 8.181596904133e-09 true resid norm >> 8.181596904190e-09 ||Ae||/||Ax|| 1.398343527570e-03 >> 0 KSP preconditioned resid norm 2.385292955090e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.385292955090e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.286979495932e-01 true resid >> norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 >> 1 KSP preconditioned resid norm 5.286979495932e-01 true resid >> norm 7.984208016570e-01 ||Ae||/||Ax|| 7.984208016570e-01 >> 54 KSP preconditioned resid norm 6.409203469073e-09 true resid norm >> 6.409203469062e-09 ||Ae||/||Ax|| 1.095417959696e-03 >> 0 KSP preconditioned resid norm 5.707419973376e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.707419973376e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 6.431212555078e-01 true resid >> norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 >> 1 KSP preconditioned resid norm 6.431212555078e-01 true resid >> norm 1.319769139913e+00 ||Ae||/||Ax|| 1.319769139913e+00 >> 55 KSP preconditioned resid norm 4.450638029919e-09 true resid norm >> 4.450638029969e-09 ||Ae||/||Ax|| 7.606731247757e-04 >> 0 KSP preconditioned resid norm 7.199725849178e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 7.199725849178e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.820447922491e-01 true resid >> norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 >> 1 KSP preconditioned resid norm 8.820447922491e-01 true resid >> norm 1.569220175230e+00 ||Ae||/||Ax|| 1.569220175230e+00 >> 56 KSP preconditioned resid norm 2.820717192827e-09 true resid norm >> 2.820717192685e-09 ||Ae||/||Ax|| 4.820980153004e-04 >> 0 KSP preconditioned resid norm 3.954316557881e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.954316557881e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.174082394688e+00 true resid >> norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 >> 1 KSP preconditioned resid norm 1.174082394688e+00 true resid >> norm 1.017969498279e+00 ||Ae||/||Ax|| 1.017969498279e+00 >> 57 KSP preconditioned resid norm 1.751635661806e-09 true resid norm >> 1.751635661897e-09 ||Ae||/||Ax|| 2.993777888545e-04 >> 0 KSP preconditioned resid norm 1.876208896020e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.876208896020e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.566896137837e+00 true resid >> norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 >> 1 KSP preconditioned resid norm 1.566896137837e+00 true resid >> norm 6.914905383075e-01 ||Ae||/||Ax|| 6.914905383075e-01 >> 58 KSP preconditioned resid norm 1.207891841472e-09 true resid norm >> 1.207891841440e-09 ||Ae||/||Ax|| 2.064447513440e-04 >> 0 KSP preconditioned resid norm 2.297261201903e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.297261201903e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.961454513785e+00 true resid >> norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 >> 1 KSP preconditioned resid norm 1.961454513785e+00 true resid >> norm 7.361082716350e-01 ||Ae||/||Ax|| 7.361082716350e-01 >> 59 KSP preconditioned resid norm 1.016539279879e-09 true resid norm >> 1.016539279925e-09 ||Ae||/||Ax|| 1.737400582368e-04 >> 0 KSP preconditioned resid norm 3.254668263088e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.254668263088e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.640547979130e+00 true resid >> norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 >> 1 KSP preconditioned resid norm 1.640547979130e+00 true resid >> norm 7.473445815641e-01 ||Ae||/||Ax|| 7.473445815641e-01 >> 60 KSP preconditioned resid norm 8.558459393246e-10 true resid norm >> 8.558459393246e-10 ||Ae||/||Ax|| 1.462754330073e-04 >> 0 KSP preconditioned resid norm 3.699188433787e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.699188433787e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 3.641405579973e+00 true resid >> norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 >> 1 KSP preconditioned resid norm 3.641405579973e+00 true resid >> norm 9.275791982055e-01 ||Ae||/||Ax|| 9.275791982055e-01 >> 61 KSP preconditioned resid norm 7.891395315495e-10 true resid norm >> 7.891395315494e-10 ||Ae||/||Ax|| 1.348744223425e-04 >> 0 KSP preconditioned resid norm 2.675881805310e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.675881805310e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.453713947810e+00 true resid >> norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 >> 1 KSP preconditioned resid norm 1.453713947810e+00 true resid >> norm 7.138367623914e-01 ||Ae||/||Ax|| 7.138367623914e-01 >> 62 KSP preconditioned resid norm 7.510552985484e-10 true resid norm >> 7.510552985213e-10 ||Ae||/||Ax|| 1.283653213221e-04 >> 0 KSP preconditioned resid norm 1.338840353148e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.338840353148e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.146220467265e+00 true resid >> norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 >> 1 KSP preconditioned resid norm 1.146220467265e+00 true resid >> norm 4.297177669038e-01 ||Ae||/||Ax|| 4.297177669038e-01 >> 63 KSP preconditioned resid norm 6.832740701908e-10 true resid norm >> 6.832740702025e-10 ||Ae||/||Ax|| 1.167806095574e-04 >> 0 KSP preconditioned resid norm 2.478031412327e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.478031412327e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.547561676965e+00 true resid >> norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 >> 1 KSP preconditioned resid norm 1.547561676965e+00 true resid >> norm 7.067319359604e-01 ||Ae||/||Ax|| 7.067319359604e-01 >> 64 KSP preconditioned resid norm 6.312416928849e-10 true resid norm >> 6.312416928420e-10 ||Ae||/||Ax|| 1.078875854989e-04 >> 0 KSP preconditioned resid norm 2.510304348429e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.510304348429e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.470709935588e+00 true resid >> norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 >> 1 KSP preconditioned resid norm 1.470709935588e+00 true resid >> norm 6.572137642951e-01 ||Ae||/||Ax|| 6.572137642951e-01 >> 65 KSP preconditioned resid norm 5.327354965414e-10 true resid norm >> 5.327354965097e-10 ||Ae||/||Ax|| 9.105156880435e-05 >> 0 KSP preconditioned resid norm 2.476662569003e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.476662569003e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.310145576275e+00 true resid >> norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 >> 1 KSP preconditioned resid norm 2.310145576275e+00 true resid >> norm 6.408918975397e-01 ||Ae||/||Ax|| 6.408918975397e-01 >> 66 KSP preconditioned resid norm 4.469580732178e-10 true resid norm >> 4.469580732741e-10 ||Ae||/||Ax|| 7.639106841575e-05 >> 0 KSP preconditioned resid norm 3.243520644840e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.243520644840e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.732155047053e+00 true resid >> norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 >> 1 KSP preconditioned resid norm 2.732155047053e+00 true resid >> norm 7.812059837441e-01 ||Ae||/||Ax|| 7.812059837441e-01 >> 67 KSP preconditioned resid norm 4.087445950732e-10 true resid norm >> 4.087445950372e-10 ||Ae||/||Ax|| 6.985987767337e-05 >> 0 KSP preconditioned resid norm 4.570320195732e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.570320195732e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.087601985141e+00 true resid >> norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 >> 1 KSP preconditioned resid norm 2.087601985141e+00 true resid >> norm 1.032266974836e+00 ||Ae||/||Ax|| 1.032266974836e+00 >> 68 KSP preconditioned resid norm 3.841153386650e-10 true resid norm >> 3.841153386559e-10 ||Ae||/||Ax|| 6.565041078654e-05 >> 0 KSP preconditioned resid norm 5.718860667569e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 5.718860667569e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.709934079219e+00 true resid >> norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 >> 1 KSP preconditioned resid norm 1.709934079219e+00 true resid >> norm 1.273174095131e+00 ||Ae||/||Ax|| 1.273174095131e+00 >> 69 KSP preconditioned resid norm 3.520264512665e-10 true resid norm >> 3.520264512287e-10 ||Ae||/||Ax|| 6.016599392193e-05 >> 0 KSP preconditioned resid norm 4.537109521149e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.537109521149e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.937349360261e+00 true resid >> norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 >> 1 KSP preconditioned resid norm 1.937349360261e+00 true resid >> norm 1.046686001054e+00 ||Ae||/||Ax|| 1.046686001054e+00 >> 70 KSP preconditioned resid norm 3.149991658898e-10 true resid norm >> 3.149991658892e-10 ||Ae||/||Ax|| 5.383753929329e-05 >> 0 KSP preconditioned resid norm 3.602956988994e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.602956988994e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.093057977304e+00 true resid >> norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 >> 1 KSP preconditioned resid norm 2.093057977304e+00 true resid >> norm 9.252238178106e-01 ||Ae||/||Ax|| 9.252238178106e-01 >> 71 KSP preconditioned resid norm 2.993160400510e-10 true resid norm >> 2.993160400417e-10 ||Ae||/||Ax|| 5.115708488107e-05 >> 0 KSP preconditioned resid norm 3.336754379842e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.336754379842e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.284284263997e+00 true resid >> norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 >> 1 KSP preconditioned resid norm 1.284284263997e+00 true resid >> norm 8.760668211902e-01 ||Ae||/||Ax|| 8.760668211902e-01 >> 72 KSP preconditioned resid norm 2.895591295761e-10 true resid norm >> 2.895591295591e-10 ||Ae||/||Ax|| 4.948949934951e-05 >> 0 KSP preconditioned resid norm 2.267067098558e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.267067098558e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.534973699953e-01 true resid >> norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 >> 1 KSP preconditioned resid norm 8.534973699953e-01 true resid >> norm 7.726740440864e-01 ||Ae||/||Ax|| 7.726740440864e-01 >> 73 KSP preconditioned resid norm 2.812425874590e-10 true resid norm >> 2.812425874308e-10 ||Ae||/||Ax|| 4.806809189165e-05 >> 0 KSP preconditioned resid norm 1.144552955812e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.144552955812e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 6.664517534844e-01 true resid >> norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 >> 1 KSP preconditioned resid norm 6.664517534844e-01 true resid >> norm 6.421695318305e-01 ||Ae||/||Ax|| 6.421695318305e-01 >> 74 KSP preconditioned resid norm 2.735484153434e-10 true resid norm >> 2.735484153084e-10 ||Ae||/||Ax|| 4.675305573021e-05 >> 0 KSP preconditioned resid norm 1.554702841568e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.554702841568e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 7.157380114110e-01 true resid >> norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 >> 1 KSP preconditioned resid norm 7.157380114110e-01 true resid >> norm 7.377660147515e-01 ||Ae||/||Ax|| 7.377660147515e-01 >> 75 KSP preconditioned resid norm 2.661574205602e-10 true resid norm >> 2.661574205227e-10 ||Ae||/||Ax|| 4.548983660049e-05 >> 0 KSP preconditioned resid norm 4.199331451693e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 4.199331451693e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 7.261245258089e-01 true resid >> norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 >> 1 KSP preconditioned resid norm 7.261245258089e-01 true resid >> norm 9.689217434547e-01 ||Ae||/||Ax|| 9.689217434547e-01 >> 76 KSP preconditioned resid norm 2.534998275631e-10 true resid norm >> 2.534998275209e-10 ||Ae||/||Ax|| 4.332648591775e-05 >> 0 KSP preconditioned resid norm 3.743395182510e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.743395182510e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.990417749799e-01 true resid >> norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 >> 1 KSP preconditioned resid norm 8.990417749799e-01 true resid >> norm 9.329106139559e-01 ||Ae||/||Ax|| 9.329106139559e-01 >> 77 KSP preconditioned resid norm 2.388049684536e-10 true resid norm >> 2.388049684532e-10 ||Ae||/||Ax|| 4.081493941813e-05 >> 0 KSP preconditioned resid norm 1.894155910280e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.894155910280e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.684190700742e-01 true resid >> norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 >> 1 KSP preconditioned resid norm 8.684190700742e-01 true resid >> norm 7.727740370555e-01 ||Ae||/||Ax|| 7.727740370555e-01 >> 78 KSP preconditioned resid norm 2.286512776556e-10 true resid norm >> 2.286512776733e-10 ||Ae||/||Ax|| 3.907953886624e-05 >> 0 KSP preconditioned resid norm 1.276207551837e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.276207551837e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 5.355779104233e-01 true resid >> norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 >> 1 KSP preconditioned resid norm 5.355779104233e-01 true resid >> norm 5.278462128508e-01 ||Ae||/||Ax|| 5.278462128508e-01 >> 79 KSP preconditioned resid norm 2.074225711250e-10 true resid norm >> 2.074225711493e-10 ||Ae||/||Ax|| 3.545127109477e-05 >> 0 KSP preconditioned resid norm 1.200062868567e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.200062868567e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 6.893976477109e-01 true resid >> norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 >> 1 KSP preconditioned resid norm 6.893976477109e-01 true resid >> norm 6.184761860600e-01 ||Ae||/||Ax|| 6.184761860600e-01 >> 80 KSP preconditioned resid norm 1.878908772984e-10 true resid norm >> 1.878908773103e-10 ||Ae||/||Ax|| 3.211304532026e-05 >> 0 KSP preconditioned resid norm 1.387129119694e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.387129119694e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.742548405622e-01 true resid >> norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 >> 1 KSP preconditioned resid norm 8.742548405622e-01 true resid >> norm 6.169338830681e-01 ||Ae||/||Ax|| 6.169338830681e-01 >> 81 KSP preconditioned resid norm 1.720957363240e-10 true resid norm >> 1.720957363027e-10 ||Ae||/||Ax|| 2.941344603009e-05 >> 0 KSP preconditioned resid norm 1.602115661552e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.602115661552e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.392172521059e-01 true resid >> norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 >> 1 KSP preconditioned resid norm 8.392172521059e-01 true resid >> norm 6.206948554932e-01 ||Ae||/||Ax|| 6.206948554932e-01 >> 82 KSP preconditioned resid norm 1.584755992063e-10 true resid norm >> 1.584755991999e-10 ||Ae||/||Ax|| 2.708558378201e-05 >> 0 KSP preconditioned resid norm 1.601197866084e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.601197866084e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 8.359479912386e-01 true resid >> norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 >> 1 KSP preconditioned resid norm 8.359479912386e-01 true resid >> norm 6.119059337506e-01 ||Ae||/||Ax|| 6.119059337506e-01 >> 83 KSP preconditioned resid norm 1.418702609113e-10 true resid norm >> 1.418702608829e-10 ||Ae||/||Ax|| 2.424751101569e-05 >> 0 KSP preconditioned resid norm 1.499465989768e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.499465989768e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.045754716297e+00 true resid >> norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 >> 1 KSP preconditioned resid norm 1.045754716297e+00 true resid >> norm 6.153015126718e-01 ||Ae||/||Ax|| 6.153015126718e-01 >> 84 KSP preconditioned resid norm 1.271329562402e-10 true resid norm >> 1.271329562130e-10 ||Ae||/||Ax|| 2.172870999918e-05 >> 0 KSP preconditioned resid norm 1.634503776014e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 1.634503776014e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.222338373864e+00 true resid >> norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 >> 1 KSP preconditioned resid norm 1.222338373864e+00 true resid >> norm 6.558960123395e-01 ||Ae||/||Ax|| 6.558960123395e-01 >> 85 KSP preconditioned resid norm 1.170830928689e-10 true resid norm >> 1.170830928467e-10 ||Ae||/||Ax|| 2.001105493065e-05 >> 0 KSP preconditioned resid norm 2.189478982761e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.189478982761e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.257863289529e+00 true resid >> norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 >> 1 KSP preconditioned resid norm 1.257863289529e+00 true resid >> norm 5.975673409262e-01 ||Ae||/||Ax|| 5.975673409262e-01 >> 86 KSP preconditioned resid norm 1.072794488701e-10 true resid norm >> 1.072794488559e-10 ||Ae||/||Ax|| 1.833548202211e-05 >> 0 KSP preconditioned resid norm 3.043239239438e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 3.043239239438e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 1.508125411857e+00 true resid >> norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 >> 1 KSP preconditioned resid norm 1.508125411857e+00 true resid >> norm 5.316504517748e-01 ||Ae||/||Ax|| 5.316504517748e-01 >> 87 KSP preconditioned resid norm 9.310419868753e-11 true resid norm >> 9.310419867003e-11 ||Ae||/||Ax|| 1.591274357860e-05 >> 0 KSP preconditioned resid norm 2.707043890291e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.707043890291e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.156895340838e+00 true resid >> norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 >> 1 KSP preconditioned resid norm 2.156895340838e+00 true resid >> norm 6.565618094970e-01 ||Ae||/||Ax|| 6.565618094970e-01 >> 88 KSP preconditioned resid norm 7.852841829285e-11 true resid norm >> 7.852841828475e-11 ||Ae||/||Ax|| 1.342154920668e-05 >> 0 KSP preconditioned resid norm 2.582289916941e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.582289916941e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.382288228383e+00 true resid >> norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 >> 1 KSP preconditioned resid norm 2.382288228383e+00 true resid >> norm 6.422921135312e-01 ||Ae||/||Ax|| 6.422921135312e-01 >> 89 KSP preconditioned resid norm 6.494099865988e-11 true resid norm >> 6.494099869486e-11 ||Ae||/||Ax|| 1.109927881590e-05 >> 0 KSP preconditioned resid norm 2.588940215193e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 0 KSP preconditioned resid norm 2.588940215193e+00 true resid >> norm 1.000000000000e+00 ||Ae||/||Ax|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 2.412871891541e+00 true resid >> norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 >> 1 KSP preconditioned resid norm 2.412871891541e+00 true resid >> norm 6.483026287919e-01 ||Ae||/||Ax|| 6.483026287919e-01 >> 90 KSP preconditioned resid norm 5.586819688621e-11 true resid norm >> 5.586819689526e-11 ||Ae||/||Ax|| 9.548616540312e-06 >> 2 SNES Function norm 5.587280798316e-11 >> SNES Object: >> type: ls >> line search variant: SNESLineSearchCubic >> alpha=0.0001, maxstep=1e+08, minlambda=1e-12 >> maximum iterations=50, maximum function evaluations=10000 >> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >> total number of linear solver iterations=107 >> total number of function evaluations=3 >> KSP Object: >> type: fgmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> right preconditioning >> PC Object: >> type: mg >> MG: type is FULL, levels=1 cycles=v >> Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- >> KSP Object:(mg_levels_0_) >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> PC Object:(mg_levels_0_) >> type: ilu >> ILU: 0 levels of fill >> ILU: factor fill ratio allocated 1 >> ILU: tolerance for zero pivot 1e-12 >> ILU: using diagonal shift to prevent zero pivot >> ILU: using diagonal shift on blocks to prevent zero pivot >> out-of-place factorization >> matrix ordering: natural >> ILU: factor fill ratio needed 1 >> Factored matrix follows >> Matrix Object: >> type=seqaij, rows=39366, cols=39366 >> package used to perform factorization: petsc >> total: nonzeros=2125764, allocated nonzeros=2125764 >> using I-node routines: found 19683 nodes, limit used >> is 5 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=39366, cols=39366 >> total: nonzeros=2125764, allocated nonzeros=2125764 >> using I-node routines: found 19683 nodes, limit used is 5 >> linear system matrix = precond matrix: >> Matrix Object: >> type=seqaij, rows=39366, cols=39366 >> total: nonzeros=2125764, allocated nonzeros=2125764 >> using I-node routines: found 19683 nodes, limit used is 5 >> >> Thanks again, >> Hal >> >> >> >> >> Barry Smith wrote: >>> Please run with >>> -ksp_monitor_true_residual -snes_monitor >>> -mg_levels_ksp_monitor_true_residua -mg_coarse_ksp_type gmres >>> -mg_coarse_ksp_monitor_true_residual -snes_view >>> and send the output for one time-step. >>> Barry >>> On Aug 26, 2009, at 2:34 PM, Hal Finkel wrote: >>>> Hello, >>>> >>>> I've constructed a evolution scheme for a time-dependent 3D PDE >>>> system using the DMMG layer. The core of the code is essentially a >>>> loop around DMMGSolve in the spirit of >>>> snes/examples/tutorials/ex29.c. If I only run with one level, then >>>> the code runs and produces sensible results. Passing the option >>>> -mg_levels_0_ksp_type bcgsl increases the solver speed by about 30%, >>>> and -mg_levels_0_pc_type fieldsplit gives me another 5% improvement. >>>> Does that say something about the structure of my problem? >>>> >>>> If I run with two levels, the coarse solve seems to work, but the >>>> fine grid's KSP never seems to complete. If I run with >>>> -mg_levels_1_ksp_monitor_true_residual, I can see the residual does >>>> not seem to be converging at all: >>>> >>>> 0 KSP preconditioned resid norm 1.979915874387e+02 true resid >>>> norm 1.889855551272e+04 ||Ae||/||Ax|| 1.889855551272e+04 >>>> 0 KSP preconditioned resid norm 3.833279709488e+06 true resid >>>> norm 3.665290817713e+08 ||Ae||/||Ax|| 3.665290817713e+08 >>>> 0 KSP preconditioned resid norm 5.080465654647e+01 true resid >>>> norm 8.284057252719e+03 ||Ae||/||Ax|| 8.284057252719e+03 >>>> 1 KSP preconditioned resid norm 6.057649668915e+00 true resid >>>> norm 5.136009463814e+02 ||Ae||/||Ax|| 5.136009463814e+02 >>>> 0 KSP preconditioned resid norm 5.511785811223e+04 true resid >>>> norm 1.055162492334e+07 ||Ae||/||Ax|| 1.055162492334e+07 >>>> 0 KSP preconditioned resid norm 2.164672340919e+01 true resid >>>> norm 2.999406329054e+04 ||Ae||/||Ax|| 2.999406329054e+04 >>>> 0 KSP preconditioned resid norm 5.275983407134e+05 true resid >>>> norm 9.196351032040e+08 ||Ae||/||Ax|| 9.196351032040e+08 >>>> 0 KSP preconditioned resid norm 3.338344860439e+01 true resid >>>> norm 5.133553722923e+03 ||Ae||/||Ax|| 5.133553722923e+03 >>>> 1 KSP preconditioned resid norm 4.306407436023e+00 true resid >>>> norm 3.361745947223e+02 ||Ae||/||Ax|| 3.361745947223e+02 >>>> ... >>>> >>>> If I run with -pc_mg_monitor I see, after the course solve, only >>>> things like: >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 1.979915874387e+02 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 3.833279709488e+06 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 5.080465654647e+01 >>>> 1 KSP Residual norm 6.057649668915e+00 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 5.511785811223e+04 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 2.164672340919e+01 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 5.275983407134e+05 >>>> Residual norms for mg_levels_1_ solve. >>>> 0 KSP Residual norm 3.338344860439e+01 >>>> 1 KSP Residual norm 4.306407436023e+00 >>>> ... >>>> >>>> running with -info shows: >>>> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >>>> computing 2-norm of preconditioned RHS >>>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >>>> hand size norm 0.0066162, current residual norm 23282.7 at iteration 0 >>>> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >>>> computing 2-norm of preconditioned RHS >>>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >>>> hand size norm 0.00287711, current residual norm 38.2205 at iteration 0 >>>> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >>>> computing 2-norm of preconditioned RHS >>>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >>>> hand size norm 0.00287711, current residual norm 809191 at iteration 0 >>>> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >>>> computing 2-norm of preconditioned RHS >>>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >>>> hand size norm 0.00970186, current residual norm 186.983 at iteration 0 >>>> [0] KSPDefaultConverged(): user has provided nonzero initial guess, >>>> computing 2-norm of preconditioned RHS >>>> [0] KSPDefaultConverged(): Linear solver is diverging. Initial right >>>> hand size norm 0.00970186, current residual norm 3.62292e+06 at >>>> iteration 0 >>>> >>>> What is a good way to debug this? >>>> >>>> No matter how I run, I've arranged the code so that the fine grid is >>>> *always* the same size; it has to be because the initial conditions >>>> are loaded from a file. >>>> >>>> In general, however, I don't understand exactly what the mg >>>> preconditioner is doing. Is there a writeup somewhere which is more >>>> verbose than the manual? In the one-level case, how are the results >>>> of the solve used to precondition the Jacobian? >>>> >>>> In an 1-processor, 1-level case, running with -snes_view gives: >>>> SNES Object: >>>> type: ls >>>> line search variant: SNESLineSearchCubic >>>> alpha=0.0001, maxstep=1e+08, minlambda=1e-12 >>>> maximum iterations=50, maximum function evaluations=10000 >>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 >>>> total number of linear solver iterations=36 >>>> total number of function evaluations=3 >>>> KSP Object: >>>> type: fgmres >>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >>>> Orthogonalization with no iterative refinement >>>> GMRES: happy breakdown tolerance 1e-30 >>>> maximum iterations=10000, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> right preconditioning >>>> PC Object: >>>> type: mg >>>> MG: type is FULL, levels=1 cycles=v >>>> Coarse gride solver -- level 0 presmooths=1 postsmooths=1 ----- >>>> KSP Object:(mg_levels_0_) >>>> type: bcgsl >>>> BCGSL: Ell = 2 >>>> BCGSL: Delta = 0 >>>> maximum iterations=1, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> PC Object:(mg_levels_0_) >>>> type: fieldsplit >>>> FieldSplit with MULTIPLICATIVE composition: total splits = >>>> 2, blocksize = 2 >>>> Solver info for each split is in the following KSP objects: >>>> Split number 0 Fields 0 >>>> KSP Object:(mg_levels_0_fieldsplit_0_) >>>> type: preonly >>>> maximum iterations=10000, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> PC Object:(mg_levels_0_fieldsplit_0_) >>>> type: ilu >>>> ILU: 0 levels of fill >>>> ILU: factor fill ratio allocated 1 >>>> ILU: tolerance for zero pivot 1e-12 >>>> ILU: using diagonal shift to prevent zero pivot >>>> ILU: using diagonal shift on blocks to prevent zero pivot >>>> out-of-place factorization >>>> matrix ordering: natural >>>> ILU: factor fill ratio needed 1 >>>> Factored matrix follows >>>> Matrix Object: >>>> type=seqaij, rows=19683, cols=19683 >>>> package used to perform factorization: petsc >>>> total: nonzeros=531441, allocated nonzeros=531441 >>>> not using I-node routines >>>> linear system matrix = precond matrix: >>>> Matrix Object: >>>> type=seqaij, rows=19683, cols=19683 >>>> total: nonzeros=531441, allocated nonzeros=531441 >>>> not using I-node routines >>>> Split number 1 Fields 1 >>>> KSP Object:(mg_levels_0_fieldsplit_1_) >>>> type: preonly >>>> maximum iterations=10000, initial guess is zero >>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>> left preconditioning >>>> PC Object:(mg_levels_0_fieldsplit_1_) >>>> type: ilu >>>> ILU: 0 levels of fill >>>> ILU: factor fill ratio allocated 1 >>>> ILU: tolerance for zero pivot 1e-12 >>>> ILU: using diagonal shift to prevent zero pivot >>>> ILU: using diagonal shift on blocks to prevent zero pivot >>>> out-of-place factorization >>>> matrix ordering: natural >>>> ILU: factor fill ratio needed 1 >>>> Factored matrix follows >>>> Matrix Object: >>>> type=seqaij, rows=19683, cols=19683 >>>> package used to perform factorization: petsc >>>> total: nonzeros=531441, allocated nonzeros=531441 >>>> not using I-node routines >>>> linear system matrix = precond matrix: >>>> Matrix Object: >>>> type=seqaij, rows=19683, cols=19683 >>>> total: nonzeros=531441, allocated nonzeros=531441 >>>> not using I-node routines >>>> linear system matrix = precond matrix: >>>> Matrix Object: >>>> type=seqaij, rows=39366, cols=39366 >>>> total: nonzeros=2125764, allocated nonzeros=2125764 >>>> using I-node routines: found 19683 nodes, limit used is 5 >>>> linear system matrix = precond matrix: >>>> Matrix Object: >>>> type=seqaij, rows=39366, cols=39366 >>>> total: nonzeros=2125764, allocated nonzeros=2125764 >>>> using I-node routines: found 19683 nodes, limit used is 5 >>>> >>>> Thank you in advance, >>>> Hal From rodrigowpa at gmail.com Thu Aug 27 12:15:57 2009 From: rodrigowpa at gmail.com (Rodrigo Araujo) Date: Thu, 27 Aug 2009 14:15:57 -0300 Subject: Questions about BLAS Message-ID: <357feb30908271015l7906309eq509a679fa236fab0@mail.gmail.com> Good afternoon, I am multiplying two matrices using the function MatMatMult_MPIAIJ_MPIAIJ, but I'd like to know if this function calls the BLAS routine that multiply matrices, and if yes ,where. Best Regards. -- Rodrigo W. Pimentel Araujo Engenharia da Computa??o UFPE -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.aunai at gmail.com Thu Aug 27 12:51:57 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Thu, 27 Aug 2009 19:51:57 +0200 Subject: VecScatterCreateToZero Message-ID: Hello, I have a problem using 'VecScatterCreateToZero'. I attach to this email a test code that illustrates the problem. I create a 2D DA then I ask for the coordinates (that have been set just before), which are returned in a global vector. I want to write the coordinates in a file, so I use 'VecScatterCreateToZero' to put all the values in sequential vector. It seems to work fine but sometimes (depending on the number of proc used) the coordinates in the sequential vectors are ordered differently. to run the code and see the problem : mpiexec -n 4 ./vecdacoord > problem.txt I have attached problem.txt too. Is someone can explain to me how I could obtain my coordinates in a natural order ? Thx -------------- next part -------------- Process [0] 0 0 0.25 0 0.5 0 0.75 0 1 0 0 0.25 0.25 0.25 0.5 0.25 0.75 0.25 Process [1] 1 0.25 0 0.5 0.25 0.5 0.5 0.5 0.75 0.5 1 0.5 Process [2] 0 0.75 0.25 0.75 0.5 0.75 0.75 0.75 1 0.75 0 1 Process [3] 0.25 1 0.5 1 0.75 1 1 1 0 0 0.25 0 0.5 0 0 0.25 0.25 0.25 0.5 0.25 0 0.5 0.25 0.5 0.5 0.5 0.75 0 1 0 0.75 0.25 1 0.25 0.75 0.5 1 0.5 0 0.75 0.25 0.75 0.5 0.75 0 1 0.25 1 0.5 1 0.75 0.75 1 0.75 0.75 1 1 1 -------------- next part -------------- A non-text attachment was scrubbed... Name: vecdacoord.c Type: text/x-csrc Size: 1946 bytes Desc: not available URL: From knepley at gmail.com Thu Aug 27 13:14:53 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Aug 2009 13:14:53 -0500 Subject: Questions about BLAS In-Reply-To: <357feb30908271015l7906309eq509a679fa236fab0@mail.gmail.com> References: <357feb30908271015l7906309eq509a679fa236fab0@mail.gmail.com> Message-ID: On Thu, Aug 27, 2009 at 12:15 PM, Rodrigo Araujo wrote: > Good afternoon, > > I am multiplying two matrices using the function MatMatMult_MPIAIJ_MPIAIJ, > but I'd like to know if this function calls the BLAS routine that multiply > matrices, and if yes ,where. No, this is for sparse matrices. Matt > > Best Regards. > > -- > Rodrigo W. Pimentel Araujo > Engenharia da Computa??o > UFPE > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 27 13:20:37 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Aug 2009 13:20:37 -0500 Subject: VecScatterCreateToZero In-Reply-To: References: Message-ID: This should be good enough http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/DA/DAGlobalToNaturalAllCreate.html Matt On Thu, Aug 27, 2009 at 12:51 PM, nicolas aunai wrote: > Hello, > > > I have a problem using 'VecScatterCreateToZero'. I attach to this > email a test code that illustrates the problem. I create a 2D DA then > I ask for the coordinates (that have been set just before), which are > returned in a global vector. I want to write the coordinates in a > file, so I use 'VecScatterCreateToZero' to put all the values in > sequential vector. It seems to work fine but sometimes (depending on > the number of proc used) the coordinates in the sequential vectors are > ordered differently. > > to run the code and see the problem : > > mpiexec -n 4 ./vecdacoord > problem.txt > > I have attached problem.txt too. > > Is someone can explain to me how I could obtain my coordinates in a > natural order ? > > > Thx > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodrigowpa at gmail.com Thu Aug 27 13:35:38 2009 From: rodrigowpa at gmail.com (Rodrigo Araujo) Date: Thu, 27 Aug 2009 15:35:38 -0300 Subject: Questions about BLAS In-Reply-To: References: <357feb30908271015l7906309eq509a679fa236fab0@mail.gmail.com> Message-ID: <357feb30908271135n2197fd4fh2c012f3c4a7b1632@mail.gmail.com> Matt, Thank you for answering. Another question. Which functions does the PETSc call the BLAS routines to multiply those matrices? Thanks. 2009/8/27 Matthew Knepley > On Thu, Aug 27, 2009 at 12:15 PM, Rodrigo Araujo wrote: > >> Good afternoon, >> >> I am multiplying two matrices using the function MatMatMult_MPIAIJ_MPIAIJ, >> but I'd like to know if this function calls the BLAS routine that multiply >> matrices, and if yes ,where. > > > No, this is for sparse matrices. > > Matt > > >> >> Best Regards. >> >> -- >> Rodrigo W. Pimentel Araujo >> Engenharia da Computa??o >> UFPE >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- Rodrigo W. Pimentel Araujo Engenharia da Computa??o UFPE -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Aug 27 13:48:13 2009 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 27 Aug 2009 13:48:13 -0500 Subject: Questions about BLAS In-Reply-To: <357feb30908271135n2197fd4fh2c012f3c4a7b1632@mail.gmail.com> References: <357feb30908271015l7906309eq509a679fa236fab0@mail.gmail.com> <357feb30908271135n2197fd4fh2c012f3c4a7b1632@mail.gmail.com> Message-ID: On Thu, Aug 27, 2009 at 1:35 PM, Rodrigo Araujo wrote: > Matt, > > Thank you for answering. Another question. Which functions does the PETSc > call the BLAS routines to multiply those matrices? Only the DENSE amtrix type uses BLAS. Matt > > Thanks. > > 2009/8/27 Matthew Knepley > >> On Thu, Aug 27, 2009 at 12:15 PM, Rodrigo Araujo wrote: >> >>> Good afternoon, >>> >>> I am multiplying two matrices using the function >>> MatMatMult_MPIAIJ_MPIAIJ, but I'd like to know if this function calls the >>> BLAS routine that multiply matrices, and if yes ,where. >> >> >> No, this is for sparse matrices. >> >> Matt >> >> >>> >>> Best Regards. >>> >>> -- >>> Rodrigo W. Pimentel Araujo >>> Engenharia da Computa??o >>> UFPE >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > Rodrigo W. Pimentel Araujo > Engenharia da Computa??o > UFPE > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.aunai at gmail.com Fri Aug 28 02:54:36 2009 From: nicolas.aunai at gmail.com (nicolas aunai) Date: Fri, 28 Aug 2009 09:54:36 +0200 Subject: VecScatterCreateToZero In-Reply-To: References: Message-ID: Hi, Hum... Yes and no. This function does indeed create a VectorScatter that allows to go from parallel to sequential and natural vectors, but the sequential vectors must but structured to receive the DA data. For example a 2D DA with 3 degree of freedom will require a specific vector with a global size equal to nx*ny*3 I want to scatter the coordinates vector obtained with the function DAGetCoordinates(), which is of size nx*ny*2. This is the reason why I was not using the DACreateNaturalVector() function and then VecScatterCreateToZero(), the natural vector being created has a wrong size. 2009/8/27 Matthew Knepley : > This should be good enough > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/DA/DAGlobalToNaturalAllCreate.html > > ???? Matt > > On Thu, Aug 27, 2009 at 12:51 PM, nicolas aunai > wrote: >> >> Hello, >> >> >> I have a problem using 'VecScatterCreateToZero'. I attach to this >> email a test code that illustrates the problem. I create a 2D DA then >> I ask for the coordinates (that have been set just before), which are >> returned in a global vector. I want to write the coordinates in a >> file, so I use 'VecScatterCreateToZero' to put all the values in >> sequential vector. It seems to work fine but sometimes (depending on >> the number of proc used) the coordinates in the sequential vectors are >> ordered differently. >> >> to run the code and see the problem : >> >> mpiexec -n 4 ./vecdacoord > problem.txt >> >> I have attached problem.txt too. >> >> Is someone can explain to me how I could obtain my coordinates in a >> natural order ? >> >> >> Thx > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > From knepley at gmail.com Fri Aug 28 05:30:09 2009 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 28 Aug 2009 05:30:09 -0500 Subject: VecScatterCreateToZero In-Reply-To: References: Message-ID: Use the coodinate DA. Matt On Fri, Aug 28, 2009 at 2:54 AM, nicolas aunai wrote: > Hi, > > Hum... Yes and no. This function does indeed create a VectorScatter > that allows to go from parallel to sequential and natural vectors, but > the sequential vectors must but structured to receive the DA data. For > example a 2D DA with 3 degree of freedom will require a specific > vector with a global size equal to nx*ny*3 > > I want to scatter the coordinates vector obtained with the function > DAGetCoordinates(), which is of size nx*ny*2. > > This is the reason why I was not using the DACreateNaturalVector() > function and then VecScatterCreateToZero(), the natural vector being > created has a wrong size. > > > > > > 2009/8/27 Matthew Knepley : > > This should be good enough > > > > > > > http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/DA/DAGlobalToNaturalAllCreate.html > > > > Matt > > > > On Thu, Aug 27, 2009 at 12:51 PM, nicolas aunai > > > wrote: > >> > >> Hello, > >> > >> > >> I have a problem using 'VecScatterCreateToZero'. I attach to this > >> email a test code that illustrates the problem. I create a 2D DA then > >> I ask for the coordinates (that have been set just before), which are > >> returned in a global vector. I want to write the coordinates in a > >> file, so I use 'VecScatterCreateToZero' to put all the values in > >> sequential vector. It seems to work fine but sometimes (depending on > >> the number of proc used) the coordinates in the sequential vectors are > >> ordered differently. > >> > >> to run the code and see the problem : > >> > >> mpiexec -n 4 ./vecdacoord > problem.txt > >> > >> I have attached problem.txt too. > >> > >> Is someone can explain to me how I could obtain my coordinates in a > >> natural order ? > >> > >> > >> Thx > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments > > is infinitely more interesting than any results to which their > experiments > > lead. > > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsjb00 at hotmail.com Mon Aug 31 12:35:17 2009 From: tsjb00 at hotmail.com (tsjb00) Date: Mon, 31 Aug 2009 17:35:17 +0000 Subject: Create With Array Message-ID: Hi! I have a quick question about the 'CreateWith Array' subroutines. Do they increase memory requirement? I am wondering if new memory space is allocated for the vector/matrix etc, or the vector/matrix is accessed through the same memory occupied by array. Thanks and have a good day! _________________________________________________________________ ????????????360??????? http://club.msn.cn/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Aug 31 12:38:20 2009 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 31 Aug 2009 12:38:20 -0500 (CDT) Subject: Create With Array In-Reply-To: References: Message-ID: No new memory is allocated. It uses the array specified by the user.. Satish On Mon, 31 Aug 2009, tsjb00 wrote: > > Hi! I have a quick question about the 'CreateWith Array' subroutines. Do they increase memory requirement? I am wondering if new memory space is allocated for the vector/matrix etc, or the vector/matrix is accessed through the same memory occupied by array. > > Thanks and have a good day! > _________________________________________________________________ > ????????????360??????? > http://club.msn.cn/