From zonexo at gmail.com Mon Oct 1 09:36:37 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 01 Oct 2012 16:36:37 +0200 Subject: [petsc-users] Using MatSetValues to add all matrix values in 1 run Message-ID: <5069AA75.5030006@gmail.com> Hi, I'm trying to insert values into my PETSc matrix using MatSetValues. I tried using: /do ijk=ijk_xyz_sta,ijk_xyz_end// // // II = ijk - 1 !Fortran shift to 0-based// //// // call MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// // //end do/ where ijk_xyz_sta/ijk_xyz_end are the starting/end index int_semi_xyz(ijk,1:7) stores the 7 column global indices semi_mat_xyz has the corresponding values. This work well but I'm trying to call MatSetValues in 1 statement, it failed. I tried: /do ijk = 1,ijk_xyz_end-ijk_xyz_sta// // // rows(ijk) = ijk_xyz_sta + ijk - 1// !to get row's global indices //// //end do// // //call MatSetValues(A_semi_xyz,ijk_xyz_end-ijk_xyz_sta,rows,7,int_semi_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),semi_mat_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),INSERT_VALUES,ierr)/ but when I use MatView, the values are in the wrong locations. May I know what's wrong? Thanks -- Yours sincerely, TAY wee-beng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Oct 1 09:39:02 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 1 Oct 2012 10:39:02 -0400 Subject: [petsc-users] Using MatSetValues to add all matrix values in 1 run In-Reply-To: <5069AA75.5030006@gmail.com> References: <5069AA75.5030006@gmail.com> Message-ID: MatSetValues() inserts logically dense blocks. You are thinking of it as inserting a bunch of arbitrary (row,col,value) but that's not how it works. Read the man page and look at any of the examples. On Mon, Oct 1, 2012 at 10:36 AM, TAY wee-beng wrote: > Hi, > > I'm trying to insert values into my PETSc matrix using MatSetValues. > > I tried using: > > *do ijk=ijk_xyz_sta,ijk_xyz_end** > ** > ** II = ijk - 1 !Fortran shift to 0-based** > ** ** > ** call > MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) > ** > ** > **end do* > > where ijk_xyz_sta/ijk_xyz_end are the starting/end index > > int_semi_xyz(ijk,1:7) stores the 7 column global indices > > semi_mat_xyz has the corresponding values. > > This work well but I'm trying to call MatSetValues in 1 statement, it > failed. I tried: > > *do ijk = 1,ijk_xyz_end-ijk_xyz_sta** > ** > ** rows(ijk) = ijk_xyz_sta + ijk - 1** !to get row's global indices > ** ** > **end do** > ** > **call > MatSetValues(A_semi_xyz,ijk_xyz_end-ijk_xyz_sta,rows,7,int_semi_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),semi_mat_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),INSERT_VALUES,ierr) > * > > but when I use MatView, the values are in the wrong locations. > > May I know what's wrong? > > Thanks > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 2 04:25:41 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 02 Oct 2012 11:25:41 +0200 Subject: [petsc-users] Using MatSetValues to add all matrix values in 1 run In-Reply-To: References: <5069AA75.5030006@gmail.com> Message-ID: <506AB315.6070504@gmail.com> On 1/10/2012 4:39 PM, Jed Brown wrote: > MatSetValues() inserts logically dense blocks. You are thinking of it > as inserting a bunch of arbitrary (row,col,value) but that's not how > it works. Read the man page and look at any of the examples. If that's the case, is the 1st method I used the best way to insert the matrix? In other words, I already have the A matrix and b vector in Fortran matrix, used for solving Ax = b. Is there a better or more efficient way of inserting the Fortran matrix into PETSC's matrix? Thanks > > On Mon, Oct 1, 2012 at 10:36 AM, TAY wee-beng > wrote: > > Hi, > > I'm trying to insert values into my PETSc matrix using MatSetValues. > > I tried using: > > /do ijk=ijk_xyz_sta,ijk_xyz_end// > // > // II = ijk - 1 !Fortran shift to 0-based// > //// > // call > MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// > // > //end do/ > > where ijk_xyz_sta/ijk_xyz_end are the starting/end index > > int_semi_xyz(ijk,1:7) stores the 7 column global indices > > semi_mat_xyz has the corresponding values. > > This work well but I'm trying to call MatSetValues in 1 statement, > it failed. I tried: > > /do ijk = 1,ijk_xyz_end-ijk_xyz_sta// > // > // rows(ijk) = ijk_xyz_sta + ijk - 1// !to get row's global > indices > //// > //end do// > // > //call > MatSetValues(A_semi_xyz,ijk_xyz_end-ijk_xyz_sta,rows,7,int_semi_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),semi_mat_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),INSERT_VALUES,ierr)/ > > but when I use MatView, the values are in the wrong locations. > > May I know what's wrong? > > Thanks > > -- > Yours sincerely, > > TAY wee-beng > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johannes.huber at unibas.ch Tue Oct 2 04:25:59 2012 From: johannes.huber at unibas.ch (Johannes Huber) Date: Tue, 2 Oct 2012 09:25:59 +0000 Subject: [petsc-users] Check Vec or Mat for validity Message-ID: <73644807AA76E34EA8A5562CA279021E05C4C9D4@urz-mbx-3.urz.unibas.ch> Hi, I'd like to check, whether for a Vec or a Mat the creation process (VecCrteate, VecSetSize, VecSetFromOptions or VecDuplicate or ...) has already been done or not, i.e. I'd like to check, wether the Vec or Mat is a valid object. Thanks, Hannes -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 2 06:07:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 2 Oct 2012 07:07:27 -0400 Subject: [petsc-users] Using MatSetValues to add all matrix values in 1 run In-Reply-To: <506AB315.6070504@gmail.com> References: <5069AA75.5030006@gmail.com> <506AB315.6070504@gmail.com> Message-ID: On Tue, Oct 2, 2012 at 5:25 AM, TAY wee-beng wrote: > On 1/10/2012 4:39 PM, Jed Brown wrote: > > MatSetValues() inserts logically dense blocks. You are thinking of it as > inserting a bunch of arbitrary (row,col,value) but that's not how it works. > Read the man page and look at any of the examples. > > > If that's the case, is the 1st method I used the best way to insert the > matrix? > Yes, inserting one row at a time is typical. > > In other words, I already have the A matrix and b vector in Fortran > matrix, used for solving Ax = b. > > Is there a better or more efficient way of inserting the Fortran matrix > into PETSC's matrix? > > Thanks > > > On Mon, Oct 1, 2012 at 10:36 AM, TAY wee-beng wrote: > >> Hi, >> >> I'm trying to insert values into my PETSc matrix using MatSetValues. >> >> I tried using: >> >> *do ijk=ijk_xyz_sta,ijk_xyz_end** >> ** >> ** II = ijk - 1 !Fortran shift to 0-based** >> ** ** >> ** call >> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >> ** >> ** >> **end do* >> >> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >> >> int_semi_xyz(ijk,1:7) stores the 7 column global indices >> >> semi_mat_xyz has the corresponding values. >> >> This work well but I'm trying to call MatSetValues in 1 statement, it >> failed. I tried: >> >> *do ijk = 1,ijk_xyz_end-ijk_xyz_sta** >> ** >> ** rows(ijk) = ijk_xyz_sta + ijk - 1** !to get row's global indices >> ** ** >> **end do** >> ** >> **call >> MatSetValues(A_semi_xyz,ijk_xyz_end-ijk_xyz_sta,rows,7,int_semi_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),semi_mat_xyz(ijk_xyz_sta+1:ijk_xyz_end,1:7),INSERT_VALUES,ierr) >> * >> >> but when I use MatView, the values are in the wrong locations. >> >> May I know what's wrong? >> >> Thanks >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 2 06:14:12 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 2 Oct 2012 07:14:12 -0400 Subject: [petsc-users] Check Vec or Mat for validity In-Reply-To: <73644807AA76E34EA8A5562CA279021E05C4C9D4@urz-mbx-3.urz.unibas.ch> References: <73644807AA76E34EA8A5562CA279021E05C4C9D4@urz-mbx-3.urz.unibas.ch> Message-ID: On Tue, Oct 2, 2012 at 5:25 AM, Johannes Huber wrote: > Hi, > I'd like to check, whether for a Vec or a Mat the creation process > (VecCrteate, VecSetSize, VecSetFromOptions or VecDuplicate or ...) has > already been done or not, i.e. I'd like to check, wether the Vec or Mat is > a valid object. > You can't check for something like VecCreate because the statement Vec X; does not initialize X, thus X _could_ point to valid memory (even possibly containing a valid classid, for example, if the last stack frame had allocated a vector and malloc has not given the memory back to the system and has not reused it yet), though it may also point to some garbage address that cannot be dereferenced without seg-faulting your application. You could, as a matter of practice, always write Vec X = PETSC_NULL; in which case you could check if (!X) { /* X has not be created yet */ } but this would be a convention specific to your code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From johannes.huber at unibas.ch Tue Oct 2 07:26:51 2012 From: johannes.huber at unibas.ch (Johannes Huber) Date: Tue, 2 Oct 2012 12:26:51 +0000 Subject: [petsc-users] Check Vec or Mat for validity In-Reply-To: References: <73644807AA76E34EA8A5562CA279021E05C4C9D4@urz-mbx-3.urz.unibas.ch>, Message-ID: <73644807AA76E34EA8A5562CA279021E05C4CA17@urz-mbx-3.urz.unibas.ch> Hi Jet, thanks, that helps. I forgot, that there is no construtor and thus what I asked for is just not possible to achieve. So I've got to go with your solution, which is fine for me. Thanks a lot, Hannes ________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Jed Brown [jedbrown at mcs.anl.gov] Sent: Tuesday, October 02, 2012 1:14 PM To: PETSc users list Subject: Re: [petsc-users] Check Vec or Mat for validity On Tue, Oct 2, 2012 at 5:25 AM, Johannes Huber > wrote: Hi, I'd like to check, whether for a Vec or a Mat the creation process (VecCrteate, VecSetSize, VecSetFromOptions or VecDuplicate or ...) has already been done or not, i.e. I'd like to check, wether the Vec or Mat is a valid object. You can't check for something like VecCreate because the statement Vec X; does not initialize X, thus X _could_ point to valid memory (even possibly containing a valid classid, for example, if the last stack frame had allocated a vector and malloc has not given the memory back to the system and has not reused it yet), though it may also point to some garbage address that cannot be dereferenced without seg-faulting your application. You could, as a matter of practice, always write Vec X = PETSC_NULL; in which case you could check if (!X) { /* X has not be created yet */ } but this would be a convention specific to your code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 2 07:35:55 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 02 Oct 2012 14:35:55 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> Message-ID: <506ADFAB.8080006@gmail.com> Hi, I have combined the momentum linear eqns involving x,y,z into 1 large matrix. The Poisson eqn is solved using HYPRE strcut format so it's not included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 procs. The log_summary is given below. I have some questions: 1. After combining the matrix, I should have only 1 PETSc matrix. Why does it says there are 4 matrix, 12 vector etc? 2. I'm looking at the stages which take the longest time. It seems that MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are these stages the reason why my code is not scaling well? What can I do to improve it? Btw, I insert matrix using: /do ijk=ijk_xyz_sta+1,ijk_xyz_end// // // II = ijk - 1// !Fortran shift to 0-based// //// // call MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// // //end do/ where ijk_xyz_sta/ijk_xyz_end are the starting/end index int_semi_xyz(ijk,1:7) stores the 7 column global indices semi_mat_xyz has the corresponding values. and I insert vectors using: call VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) Thanks! / / Yours sincerely, TAY wee-beng On 30/9/2012 11:30 PM, Jed Brown wrote: > > You can measure the time spent in Hypre via PCApply and PCSetUp, but > you can't get finer grained integrated profiling because it was not > set up that way. > > On Sep 30, 2012 3:26 PM, "TAY wee-beng" > wrote: > > On 27/9/2012 1:44 PM, Matthew Knepley wrote: >> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng > > wrote: >> >> Hi, >> >> I'm doing a log summary for my 3d cfd code. I have some >> questions: >> >> 1. if I'm solving 3 linear equations using ksp, is the result >> given in the log summary the total of the 3 linear eqns' >> performance? How can I get the performance for each >> individual eqn? >> >> >> Use logging stages: >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >> >> 2. If I run my code for 10 time steps, does the log summary >> gives the total or avg performance/ratio? >> >> >> Total. >> >> 3. Besides PETSc, I'm also using HYPRE's native geometric MG >> (Struct) to solve my Cartesian's grid CFD poisson eqn. Is >> there any way I can use PETSc's log summary to get HYPRE's >> performance? If I use boomerAMG thru PETSc, can I get its >> performance? >> >> >> If you mean flops, only if you count them yourself and tell PETSc >> using >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >> >> This is the disadvantage of using packages that do not properly >> monitor things :) >> >> Matt > So u mean if I use boomerAMG thru PETSc, there is no proper way of > evaluating its performance, beside using PetscLogFlops? >> >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-10 with 96 processors, by wtay Tue Oct 2 14:26:10 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 2.035e+03 1.00999 2.026e+03 Objects: 2.700e+01 1.00000 2.700e+01 Flops: 1.096e+10 1.15860 1.093e+10 1.049e+12 Flops/sec: 5.437e+06 1.16981 5.397e+06 5.181e+08 MPI Messages: 2.000e+02 2.00000 1.979e+02 1.900e+04 MPI Message Lengths: 2.598e+08 2.00000 1.299e+06 2.468e+10 MPI Reductions: 7.590e+02 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.0257e+03 100.0% 1.0495e+12 100.0% 1.900e+04 100.0% 1.299e+06 100.0% 7.580e+02 99.9% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage MatMult 98 1.0 2.1910e+01 1.8 2.51e+09 1.2 1.9e+04 1.3e+06 0.0e+00 1 23 98 99 0 1 23 98 99 0 10990 MatSolve 147 1.0 2.0458e+01 1.6 3.68e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 34 0 0 0 1 34 0 0 0 17205 MatLUFactorNum 49 1.0 2.6876e+01 1.4 2.03e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 19 0 0 0 1 19 0 0 0 7245 MatILUFactorSym 1 1.0 5.6867e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 49 1.0 5.1938e+0144.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.8e+01 1 0 0 0 13 1 0 0 0 13 0 MatAssemblyEnd 49 1.0 1.3391e+01 2.1 0.00e+00 0.0 3.8e+02 3.3e+05 8.0e+00 0 0 2 1 1 0 0 2 1 1 0 MatGetRowIJ 1 1.0 1.2875e-05 6.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 8.2746e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecDot 98 1.0 7.3859e+00 6.4 3.91e+08 1.1 0.0e+00 0.0e+00 9.8e+01 0 4 0 0 13 0 4 0 0 13 5067 VecDotNorm2 49 1.0 7.2898e+00 4.8 7.81e+08 1.1 0.0e+00 0.0e+00 1.5e+02 0 7 0 0 19 0 7 0 0 19 10269 VecNorm 98 1.0 1.2824e+0115.4 3.91e+08 1.1 0.0e+00 0.0e+00 9.8e+01 0 4 0 0 13 0 4 0 0 13 2918 VecCopy 98 1.0 1.8141e+00 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 295 1.0 3.0051e+00 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 98 1.0 4.0675e+00 1.7 7.81e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 7 0 0 0 0 7 0 0 0 18403 VecWAXPY 98 1.0 3.3849e+00 1.5 3.91e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 11057 VecAssemblyBegin 98 1.0 2.2710e+01167.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 1 0 0 0 39 1 0 0 0 39 0 VecAssemblyEnd 98 1.0 5.5361e-04 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 98 1.0 4.5025e-01 3.0 0.00e+00 0.0 1.9e+04 1.3e+06 0.0e+00 0 0 98 99 0 0 0 98 99 0 0 VecScatterEnd 98 1.0 8.9157e+0019.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 98 1.0 2.0951e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 49 1.0 8.0828e+01 1.0 1.10e+10 1.2 1.9e+04 1.3e+06 3.5e+02 4100 98 99 46 4100 98 99 46 12984 PCSetUp 98 1.0 2.7532e+01 1.4 2.03e+09 1.2 0.0e+00 0.0e+00 5.0e+00 1 19 0 0 1 1 19 0 0 1 7072 PCSetUpOnBlocks 49 1.0 2.7531e+01 1.4 2.03e+09 1.2 0.0e+00 0.0e+00 3.0e+00 1 19 0 0 0 1 19 0 0 0 7072 PCApply 147 1.0 2.1493e+01 1.6 3.68e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 34 0 0 0 1 34 0 0 0 16376 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 4 4 593048052 0 Vector 12 12 128863184 0 Vector Scatter 1 1 1060 0 Krylov Solver 2 2 2168 0 Index Set 5 5 8633616 0 Preconditioner 2 2 1800 0 Viewer 1 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000372219 Average time for zero size MPI_Send(): 2.28832e-05 #PETSc Option Table entries: -log_summary #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries From jedbrown at mcs.anl.gov Tue Oct 2 07:43:00 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 2 Oct 2012 08:43:00 -0400 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506ADFAB.8080006@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> Message-ID: On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: > Hi, > > I have combined the momentum linear eqns involving x,y,z into 1 large > matrix. The Poisson eqn is solved using HYPRE strcut format so it's not > included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 > procs. The log_summary is given below. I have some questions: > > 1. After combining the matrix, I should have only 1 PETSc matrix. Why does > it says there are 4 matrix, 12 vector etc? > They are part of preconditioning. Are you sure you're using Hypre for this? It looks like you are using bjacobi/ilu. > > 2. I'm looking at the stages which take the longest time. It seems that > MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high > ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are > these stages the reason why my code is not scaling well? What can I do to > improve it? > 3/4 of the solve time is evenly balanced between MatMult, MatSolve, MatLUFactorNumeric, and VecNorm+VecDot. The high VecAssembly time might be due to generating a lot of entries off-process? In any case, this looks like an _extremely_ slow network, perhaps it's misconfigured? > > Btw, I insert matrix using: > > *do ijk=ijk_xyz_sta+1,ijk_xyz_end** > ** > ** II = ijk - 1** !Fortran shift to 0-based** > ** ** > ** call > MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) > ** > ** > **end do* > > where ijk_xyz_sta/ijk_xyz_end are the starting/end index > > int_semi_xyz(ijk,1:7) stores the 7 column global indices > > semi_mat_xyz has the corresponding values. > > and I insert vectors using: > > call > VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) > > Thanks! > > * > * > > Yours sincerely, > > TAY wee-beng > > On 30/9/2012 11:30 PM, Jed Brown wrote: > > You can measure the time spent in Hypre via PCApply and PCSetUp, but you > can't get finer grained integrated profiling because it was not set up that > way. > On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: > >> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >> >> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >> >>> Hi, >>> >>> I'm doing a log summary for my 3d cfd code. I have some questions: >>> >>> 1. if I'm solving 3 linear equations using ksp, is the result given in >>> the log summary the total of the 3 linear eqns' performance? How can I get >>> the performance for each individual eqn? >>> >> >> Use logging stages: >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >> >> >>> 2. If I run my code for 10 time steps, does the log summary gives the >>> total or avg performance/ratio? >>> >> >> Total. >> >> >>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) to >>> solve my Cartesian's grid CFD poisson eqn. Is there any way I can use >>> PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru >>> PETSc, can I get its performance? >> >> >> If you mean flops, only if you count them yourself and tell PETSc using >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >> >> This is the disadvantage of using packages that do not properly monitor >> things :) >> >> Matt >> >> >> So u mean if I use boomerAMG thru PETSc, there is no proper way of >> evaluating its performance, beside using PetscLogFlops? >> >> >>> -- >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 2 16:02:20 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 02 Oct 2012 23:02:20 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> Message-ID: <506B565C.7050302@gmail.com> On 27/9/2012 1:44 PM, Matthew Knepley wrote: > On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng > wrote: > > Hi, > > I'm doing a log summary for my 3d cfd code. I have some questions: > > 1. if I'm solving 3 linear equations using ksp, is the result > given in the log summary the total of the 3 linear eqns' > performance? How can I get the performance for each individual eqn? > > > Use logging stages: > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html I tried to follow the steps from the manual for my Fortan code using: /call PetscInitialize(PETSC_NULL_CHARACTER,ierr)// // //... 1st stage - solving momentum eqn//- run 1st logging // //call PetscLogStagePush(1,ierr)// // //... 2nd stag - solve poisson eqn//- run 2nd logging // //call PetscLogStagePop//(ierr)// // //...// // //call PetscFinalize(ierr)/ but then I got the error: Argument out of range! [33]PETSC ERROR: Invalid stage 1 should be in [0,1)! ... How should I use the syntax to allow 2 stages logging? Tks! > 2. If I run my code for 10 time steps, does the log summary gives > the total or avg performance/ratio? > > > Total. > > 3. Besides PETSc, I'm also using HYPRE's native geometric MG > (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there > any way I can use PETSc's log summary to get HYPRE's performance? > If I use boomerAMG thru PETSc, can I get its performance? > > > If you mean flops, only if you count them yourself and tell PETSc > using > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html > > This is the disadvantage of using packages that do not properly > monitor things :) > > Matt > > > -- > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 2 16:55:50 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 02 Oct 2012 23:55:50 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> Message-ID: <506B62E6.8040604@gmail.com> On 27/9/2012 1:44 PM, Matthew Knepley wrote: > On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng > wrote: > > Hi, > > I'm doing a log summary for my 3d cfd code. I have some questions: > > 1. if I'm solving 3 linear equations using ksp, is the result > given in the log summary the total of the 3 linear eqns' > performance? How can I get the performance for each individual eqn? > > > Use logging stages: > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html I retried using : PetscLogStage stage call PetscInitialize(PETSC_NULL_CHARACTER,ierr) call PetscLogStageRegister("momentum",stage,ierr) call PetscLogStagePush(stage,ierr) / //... 1st stage - solving momentum eqn//- run 1st logging/ call PetscLogStagePop(ierr) call PetscLogStageRegister("poisson",stage,ierr) call PetscLogStagePush(stage,ierr) /... 2nd stage - solve poisson eqn//- run 2nd logging/ call PetscLogStagePop(ierr) call PetscFinalize(ierr) I have attached the log_summary results. Turns out that there are 3 stages - 0,1 (momentum),2 (poisson). Shouldn't I be getting 2 stages instead? How can I correct it? Or it doesn't matter? > > 2. If I run my code for 10 time steps, does the log summary gives > the total or avg performance/ratio? > > > Total. > > 3. Besides PETSc, I'm also using HYPRE's native geometric MG > (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there > any way I can use PETSc's log summary to get HYPRE's performance? > If I use boomerAMG thru PETSc, can I get its performance? > > > If you mean flops, only if you count them yourself and tell PETSc > using > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html > > This is the disadvantage of using packages that do not properly > monitor things :) > > Matt > > > -- > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-10 with 48 processors, by wtay Tue Oct 2 23:41:57 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 1.181e+02 1.01842 1.171e+02 Objects: 3.870e+02 1.00000 3.870e+02 Flops: 2.920e+09 1.25508 2.551e+09 1.224e+11 Flops/sec: 2.475e+07 1.24073 2.179e+07 1.046e+09 MPI Messages: 1.075e+04 3.04562 8.944e+03 4.293e+05 MPI Message Lengths: 2.917e+08 2.17986 3.148e+04 1.351e+10 MPI Reductions: 1.590e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 8.2603e+01 70.5% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+01 1.3% 1: poisson: 2.0226e+01 17.3% 6.7061e+10 54.8% 2.461e+05 57.3% 1.799e+04 57.1% 1.279e+03 80.4% 2: momentum: 1.4257e+01 12.2% 5.5380e+10 45.2% 1.832e+05 42.7% 1.349e+04 42.9% 2.900e+02 18.2% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 889 1.0 9.2293e+00 1.3 9.02e+08 1.3 1.6e+05 3.5e+04 0.0e+00 7 30 37 42 0 40 54 65 73 0 3927 MatMultAdd 129 1.0 6.0007e-01 2.2 6.14e+07 1.5 1.5e+04 9.1e+03 0.0e+00 0 2 3 1 0 2 3 6 2 0 3673 MatMultTranspose 129 1.0 1.1698e+00 3.7 6.14e+07 1.5 1.5e+04 9.1e+03 0.0e+00 0 2 3 1 0 3 3 6 2 0 1884 MatSolve 86 0.0 1.3154e-02 0.0 9.23e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 702 MatLUFactorSym 1 1.0 2.6720e-03167.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 8.1620e-031006.9 5.33e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 653 MatConvert 3 1.0 4.8410e-02 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 9 1.0 4.9452e-02 2.6 4.65e+06 1.4 5.8e+02 3.3e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 3655 MatAssemblyBegin 50 1.0 2.3006e+00 3.4 0.00e+00 0.0 1.6e+03 6.5e+03 5.4e+01 1 0 0 0 3 7 0 1 0 4 0 MatAssemblyEnd 50 1.0 6.2769e-01 1.2 0.00e+00 0.0 9.0e+03 4.3e+03 1.4e+02 1 0 2 0 9 3 0 4 1 11 0 MatGetRow 311990 1.2 1.7204e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 0.0 1.6093e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 5.0211e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 3 1.0 6.2435e-01 1.1 0.00e+00 0.0 3.1e+04 1.8e+04 3.8e+02 1 0 7 4 24 3 0 13 7 30 0 MatAXPY 3 1.0 1.9019e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 3 1.0 4.3979e-01 1.0 3.24e+06 1.3 4.1e+03 1.4e+04 7.2e+01 0 0 1 0 5 2 0 2 1 6 294 MatMatMultSym 3 1.0 3.5735e-01 1.1 0.00e+00 0.0 3.5e+03 1.1e+04 6.6e+01 0 0 1 0 4 2 0 1 1 5 0 MatMatMultNum 3 1.0 9.2986e-02 1.2 3.24e+06 1.3 5.8e+02 3.3e+04 6.0e+00 0 0 0 0 0 0 0 0 0 0 1393 MatPtAP 3 1.0 1.1116e+00 1.0 1.13e+08 2.5 8.8e+03 2.6e+04 8.1e+01 1 2 2 2 5 5 4 4 3 6 2345 MatPtAPSymbolic 3 1.0 7.1388e-01 1.1 0.00e+00 0.0 7.9e+03 2.1e+04 7.5e+01 1 0 2 1 5 3 0 3 2 6 0 MatPtAPNumeric 3 1.0 4.0797e-01 1.1 1.13e+08 2.5 8.9e+02 6.3e+04 6.0e+00 0 2 0 0 0 2 4 0 1 0 6389 MatTrnMatMult 3 1.0 3.2513e+00 1.0 1.87e+08 5.1 3.4e+03 2.1e+05 8.7e+01 3 6 1 5 5 16 11 1 9 7 2349 MatGetLocalMat 15 1.0 1.3235e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0 MatGetBrAoCol 9 1.0 1.8796e-01 2.9 0.00e+00 0.0 4.1e+03 4.7e+04 1.2e+01 0 0 1 1 1 1 0 2 2 1 0 MatGetSymTrans 6 1.0 1.8284e-02 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 73 1.0 3.2654e-01 2.3 3.44e+07 1.2 0.0e+00 0.0e+00 7.3e+01 0 1 0 0 5 1 2 0 0 6 4578 KSPSetUp 9 1.0 4.6377e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 1 0 0 0 0 1 0 KSPSolve 1 1.0 1.9749e+01 1.0 1.56e+09 1.2 2.5e+05 3.1e+04 1.3e+03 17 55 57 57 79 98100100100 99 3396 VecDot 42 1.0 9.3734e-01 2.9 1.16e+07 1.1 0.0e+00 0.0e+00 4.2e+01 0 0 0 0 3 2 1 0 0 3 553 VecDotNorm2 21 1.0 9.9729e-01 3.2 2.32e+07 1.1 0.0e+00 0.0e+00 6.3e+01 0 1 0 0 4 3 2 0 0 5 1040 VecMDot 73 1.0 2.9734e-01 4.0 1.72e+07 1.2 0.0e+00 0.0e+00 7.3e+01 0 1 0 0 5 1 1 0 0 6 2514 VecNorm 141 1.0 6.8643e-01 3.4 9.56e+06 1.2 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 9 2 1 0 0 11 614 VecScale 635 1.0 8.8242e-02 3.7 2.86e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 14090 VecCopy 177 1.0 9.4576e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 660 1.0 7.7769e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1078 1.0 3.0908e-01 2.2 1.08e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 1 7 0 0 0 15168 VecAYPX 1032 1.0 6.9673e-01 3.5 6.71e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 2 4 0 0 0 4193 VecAXPBYCZ 42 1.0 2.4312e-01 4.8 2.32e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 2 0 0 0 4267 VecWAXPY 42 1.0 2.2274e-01 4.6 1.16e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 2329 VecMAXPY 119 1.0 8.7651e-02 3.4 2.03e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 10078 VecAssemblyBegin 115 1.0 3.7941e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.4e+02 0 0 0 0 22 1 0 0 0 27 0 VecAssemblyEnd 115 1.0 4.2558e-04 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 807 1.0 6.0901e-01 2.3 4.20e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 2 3 0 0 0 3001 VecScatterBegin 1266 1.0 2.3614e-01 2.4 0.00e+00 0.0 2.3e+05 3.0e+04 0.0e+00 0 0 53 50 0 1 0 92 87 0 0 VecScatterEnd 1266 1.0 7.4624e+00 5.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 18 0 0 0 0 0 VecSetRandom 3 1.0 8.0581e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 119 1.0 5.9730e-01 5.4 5.23e+06 1.2 0.0e+00 0.0e+00 1.2e+02 0 0 0 0 7 2 0 0 0 9 376 PCSetUp 2 1.0 7.8047e+00 1.0 3.19e+08 1.4 6.3e+04 3.2e+04 1.0e+03 7 11 15 15 63 39 20 26 26 78 1759 PCSetUpOnBlocks 43 1.0 1.1463e-0269.3 5.33e+06 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 465 PCApply 43 1.0 9.7608e+00 1.2 1.17e+09 1.3 1.8e+05 2.9e+04 1.4e+02 8 38 42 38 9 46 69 73 67 11 4774 PCGAMGgraph_AGG 3 1.0 9.0476e-01 1.0 3.24e+06 1.3 1.5e+03 1.8e+04 5.7e+01 1 0 0 0 4 4 0 1 0 4 143 PCGAMGcoarse_AGG 3 1.0 4.0595e+00 1.0 1.87e+08 5.1 3.8e+04 3.8e+04 5.3e+02 3 6 9 11 33 20 11 16 19 41 1881 PCGAMGProl_AGG 3 1.0 5.8402e-01 1.0 0.00e+00 0.0 4.2e+03 2.0e+04 8.4e+01 0 0 1 1 5 3 0 2 1 7 0 PCGAMGPOpt_AGG 3 1.0 1.1830e+00 1.0 8.01e+07 1.2 9.9e+03 2.5e+04 1.6e+02 1 3 2 2 10 6 5 4 3 12 2833 --- Event Stage 2: momentum MatMult 861 1.0 9.5562e+00 1.3 8.82e+08 1.3 1.5e+05 3.6e+04 0.0e+00 7 29 36 41 0 59 64 84 95 0 3713 MatMultAdd 129 1.0 6.3861e-01 2.1 6.14e+07 1.5 1.5e+04 9.1e+03 0.0e+00 0 2 3 1 0 3 4 8 2 0 3452 MatMultTranspose 129 1.0 1.2222e+00 3.6 6.14e+07 1.5 1.5e+04 9.1e+03 0.0e+00 1 2 3 1 0 4 4 8 2 0 1803 MatSolve 8929.7 7.7899e-02 2.1 2.46e+07 2.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 8874 MatLUFactorNum 1 1.0 1.3516e-01 2.0 8.40e+06 1.3 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 2759 MatILUFactorSym 1 1.0 1.2454e-01 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatAssemblyBegin 1 1.0 3.8949e-011845.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 2 0 0 0 1 0 MatAssemblyEnd 1 1.0 2.3289e-01 1.2 0.00e+00 0.0 1.9e+02 1.0e+05 8.0e+00 0 0 0 0 1 1 0 0 0 3 0 MatGetRowIJ 1 1.0 5.9605e-06 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.7303e-02 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 1 0 KSPGMRESOrthog 43 1.0 1.7178e-02 2.5 5.41e+04 0.0 0.0e+00 0.0e+00 4.3e+01 0 0 0 0 3 0 0 0 0 15 3 KSPSetUp 2 1.0 5.8122e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 2 1.0 1.3166e+01 1.0 1.36e+09 1.3 1.8e+05 3.2e+04 2.7e+02 11 45 43 43 17 92100100100 92 4206 VecDot 44 1.0 8.5812e-01 2.4 1.32e+07 1.1 0.0e+00 0.0e+00 4.4e+01 1 0 0 0 3 4 1 0 0 15 691 VecDotNorm2 22 1.0 9.6094e-01 2.1 2.65e+07 1.1 0.0e+00 0.0e+00 6.6e+01 1 1 0 0 4 5 2 0 0 23 1234 VecMDot 43 1.0 1.6975e-02 2.6 2.70e+04 0.0 0.0e+00 0.0e+00 4.3e+01 0 0 0 0 3 0 0 0 0 15 2 VecNorm 110 1.0 7.7599e-01 2.9 7.78e+06 1.2 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 7 3 1 0 0 38 446 VecScale 602 1.0 9.0100e-02 4.2 2.69e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 12970 VecCopy 176 1.0 1.0720e-01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 612 1.0 8.4964e-02 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1075 1.0 3.5763e-01 3.1 1.07e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 2 8 0 0 0 13071 VecAYPX 1032 1.0 6.9883e-01 4.0 6.71e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 3 5 0 0 0 4181 VecAXPBYCZ 44 1.0 2.5807e-01 4.7 2.65e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 2 0 0 0 4595 VecWAXPY 44 1.0 2.3328e-01 4.7 1.32e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 2541 VecMAXPY 86 1.0 2.7514e-04 6.2 5.42e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 197 VecAssemblyBegin 4 1.0 1.2141e-0134.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 4 0 VecAssemblyEnd 4 1.0 4.1008e-0510.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 774 1.0 8.1749e-01 3.9 4.02e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 3 3 0 0 0 2144 VecScatterBegin 1119 1.0 2.4684e-01 3.1 0.00e+00 0.0 1.8e+05 3.2e+04 0.0e+00 0 0 43 43 0 1 0100100 0 0 VecScatterEnd 1119 1.0 8.0261e+00 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 28 0 0 0 0 0 VecNormalize 86 1.0 4.9263e-0194.5 8.13e+04 0.0 0.0e+00 0.0e+00 8.6e+01 0 0 0 0 5 2 0 0 0 30 0 PCSetUp 2 1.0 2.6639e-01 2.1 8.40e+06 1.3 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 1 1 0 0 2 1400 PCSetUpOnBlocks 44 1.0 2.6618e-01 2.1 8.40e+06 1.3 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 1 1 0 0 1 1401 PCApply 46 1.0 1.0259e+01 1.1 1.18e+09 1.3 1.8e+05 2.9e+04 1.3e+02 8 39 42 38 8 69 85 98 89 44 4609 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 6 40 183988644 0 Krylov Solver 2 8 26224 0 Vector 4 54 42591168 0 Vector Scatter 0 8 8480 0 Index Set 0 8 1664536 0 Preconditioner 0 8 7988 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 84 51 118292312 0 Matrix Coarsen 3 3 1860 0 Krylov Solver 8 3 90384 0 Vector 170 129 36025624 0 Vector Scatter 22 15 15900 0 Index Set 55 50 424808 0 Preconditioner 9 3 2592 0 PetscRandom 3 3 1848 0 --- Event Stage 2: momentum Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 205792 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 3.9196e-05 Average time for zero size MPI_Send(): 2.75373e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries From bsmith at mcs.anl.gov Tue Oct 2 17:44:22 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 2 Oct 2012 17:44:22 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506B62E6.8040604@gmail.com> References: <5064051D.1060903@gmail.com> <506B62E6.8040604@gmail.com> Message-ID: <91CD4EBF-E5CD-454F-BD8C-905783A097EF@mcs.anl.gov> There will always be the "initial" stage for anything that happens before the first stage push(). Not much should be happening in it however. Barry On Oct 2, 2012, at 4:55 PM, TAY wee-beng wrote: > On 27/9/2012 1:44 PM, Matthew Knepley wrote: >> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >> Hi, >> >> I'm doing a log summary for my 3d cfd code. I have some questions: >> >> 1. if I'm solving 3 linear equations using ksp, is the result given in the log summary the total of the 3 linear eqns' performance? How can I get the performance for each individual eqn? >> >> Use logging stages: http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html > > I retried using : > > PetscLogStage stage > > call PetscInitialize(PETSC_NULL_CHARACTER,ierr) > > call PetscLogStageRegister("momentum",stage,ierr) > > call PetscLogStagePush(stage,ierr) > > ... 1st stage - solving momentum eqn - run 1st logging > > call PetscLogStagePop(ierr) > > call PetscLogStageRegister("poisson",stage,ierr) > > call PetscLogStagePush(stage,ierr) > > ... 2nd stage - solve poisson eqn - run 2nd logging > > call PetscLogStagePop(ierr) > > call PetscFinalize(ierr) > > I have attached the log_summary results. Turns out that there are 3 stages - 0,1 (momentum),2 (poisson). Shouldn't I be getting 2 stages instead? How can I correct it? Or it doesn't matter? >> >> 2. If I run my code for 10 time steps, does the log summary gives the total or avg performance/ratio? >> >> Total. >> >> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there any way I can use PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru PETSc, can I get its performance? >> >> If you mean flops, only if you count them yourself and tell PETSc using http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >> >> This is the disadvantage of using packages that do not properly monitor things :) >> >> Matt >> >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > From irving at naml.us Tue Oct 2 18:48:58 2012 From: irving at naml.us (Geoffrey Irving) Date: Tue, 2 Oct 2012 16:48:58 -0700 Subject: [petsc-users] question about MatXAIJSetPreallocation Message-ID: The dnnz argument to MatXAIJSetPreallocation is described as dnnz - number of nonzero blocks per block row of diagonal part of parallel matrix To me, this implies that if the matrix has size m and the block size is bs, dnnz should have size m/bs. However, the loop inside the routine looks like for (i=0; i References: Message-ID: On Tue, Oct 2, 2012 at 4:48 PM, Geoffrey Irving wrote: > The dnnz argument to MatXAIJSetPreallocation is described as > > dnnz - number of nonzero blocks per block row of diagonal part of > parallel matrix > > To me, this implies that if the matrix has size m and the block size > is bs, dnnz should have size m/bs. However, the loop inside the > routine looks like > > for (i=0; i if (dnnz) sdnnz[i] = dnnz[i/bs] * bs; > if (onnz) sonnz[i] = onnz[i/bs] * bs; > } > > which implies that dnnz should have m entries. Is this a bug, or am I > misinterpreting the meaning of dnnz? It looks like the loop hasn't changed since the beginning of MatXAIJSetPreallocation, but the comment changed from dnnz - number of nonzeros per row of diagonal block of parallel matrix to dnnz - number of nonzero blocks per block row of diagonal part of parallel matrix in commit abccccb5e4359792067c7bef9b175a0511adf626 Author: Jed Brown Date: Thu Mar 1 11:51:58 2012 -0600 Remove the maximum nonzero arguments from MatXAIJSetPreallocation() I'll change my code to match the first comment and see if it works. Geoffrey From jedbrown at mcs.anl.gov Tue Oct 2 21:12:14 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 2 Oct 2012 22:12:14 -0400 Subject: [petsc-users] question about MatXAIJSetPreallocation In-Reply-To: References: Message-ID: Oh noes, that's not what I ever intended. That code is creating an array that is much longer than it should be. The logic for setting block size was also changed in the spring and evidently not propagated to this routine so I added the PetscLayoutSetUp() calls that ensure local size is divisible by block size. I think this patch fixes both issues: http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/4e499e21f7a2 On Tue, Oct 2, 2012 at 7:57 PM, Geoffrey Irving wrote: > On Tue, Oct 2, 2012 at 4:48 PM, Geoffrey Irving wrote: > > The dnnz argument to MatXAIJSetPreallocation is described as > > > > dnnz - number of nonzero blocks per block row of diagonal part of > > parallel matrix > > > > To me, this implies that if the matrix has size m and the block size > > is bs, dnnz should have size m/bs. However, the loop inside the > > routine looks like > > > > for (i=0; i > if (dnnz) sdnnz[i] = dnnz[i/bs] * bs; > > if (onnz) sonnz[i] = onnz[i/bs] * bs; > > } > > > > which implies that dnnz should have m entries. Is this a bug, or am I > > misinterpreting the meaning of dnnz? > > It looks like the loop hasn't changed since the beginning of > MatXAIJSetPreallocation, but the comment changed from > > dnnz - number of nonzeros per row of diagonal block of parallel matrix > > to > > dnnz - number of nonzero blocks per block row of diagonal part of > parallel matrix > > in > > commit abccccb5e4359792067c7bef9b175a0511adf626 > Author: Jed Brown > Date: Thu Mar 1 11:51:58 2012 -0600 > > Remove the maximum nonzero arguments from MatXAIJSetPreallocation() > > I'll change my code to match the first comment and see if it works. > > Geoffrey > -------------- next part -------------- An HTML attachment was scrubbed... URL: From irving at naml.us Tue Oct 2 21:14:36 2012 From: irving at naml.us (Geoffrey Irving) Date: Tue, 2 Oct 2012 19:14:36 -0700 Subject: [petsc-users] question about MatXAIJSetPreallocation In-Reply-To: References: Message-ID: Thanks, that's what I ended up doing to fix my local version as well (and forgot to send a final email). Geoffrey On Tue, Oct 2, 2012 at 7:12 PM, Jed Brown wrote: > Oh noes, that's not what I ever intended. That code is creating an array > that is much longer than it should be. > > The logic for setting block size was also changed in the spring and > evidently not propagated to this routine so I added the PetscLayoutSetUp() > calls that ensure local size is divisible by block size. > > I think this patch fixes both issues: > > http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/4e499e21f7a2 > > > On Tue, Oct 2, 2012 at 7:57 PM, Geoffrey Irving wrote: >> >> On Tue, Oct 2, 2012 at 4:48 PM, Geoffrey Irving wrote: >> > The dnnz argument to MatXAIJSetPreallocation is described as >> > >> > dnnz - number of nonzero blocks per block row of diagonal part of >> > parallel matrix >> > >> > To me, this implies that if the matrix has size m and the block size >> > is bs, dnnz should have size m/bs. However, the loop inside the >> > routine looks like >> > >> > for (i=0; i> > if (dnnz) sdnnz[i] = dnnz[i/bs] * bs; >> > if (onnz) sonnz[i] = onnz[i/bs] * bs; >> > } >> > >> > which implies that dnnz should have m entries. Is this a bug, or am I >> > misinterpreting the meaning of dnnz? >> >> It looks like the loop hasn't changed since the beginning of >> MatXAIJSetPreallocation, but the comment changed from >> >> dnnz - number of nonzeros per row of diagonal block of parallel matrix >> >> to >> >> dnnz - number of nonzero blocks per block row of diagonal part of >> parallel matrix >> >> in >> >> commit abccccb5e4359792067c7bef9b175a0511adf626 >> Author: Jed Brown >> Date: Thu Mar 1 11:51:58 2012 -0600 >> >> Remove the maximum nonzero arguments from >> MatXAIJSetPreallocation() >> >> I'll change my code to match the first comment and see if it works. >> >> Geoffrey > > From zonexo at gmail.com Wed Oct 3 08:08:46 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 03 Oct 2012 15:08:46 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> Message-ID: <506C38DE.7030107@gmail.com> On 2/10/2012 2:43 PM, Jed Brown wrote: > On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng > wrote: > > Hi, > > I have combined the momentum linear eqns involving x,y,z into 1 > large matrix. The Poisson eqn is solved using HYPRE strcut format > so it's not included. I run the code for 50 timesteps (hence 50 > kspsolve) using 96 procs. The log_summary is given below. I have > some questions: > > 1. After combining the matrix, I should have only 1 PETSc matrix. > Why does it says there are 4 matrix, 12 vector etc? > > > They are part of preconditioning. Are you sure you're using Hypre for > this? It looks like you are using bjacobi/ilu. > > > 2. I'm looking at the stages which take the longest time. It seems > that MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd > have very high ratios. The ratios of some others are also not too > good (~ 1.6 - 2). So are these stages the reason why my code is > not scaling well? What can I do to improve it? > > > 3/4 of the solve time is evenly balanced between MatMult, MatSolve, > MatLUFactorNumeric, and VecNorm+VecDot. > > The high VecAssembly time might be due to generating a lot of entries > off-process? > > In any case, this looks like an _extremely_ slow network, perhaps it's > misconfigured? My cluster is configured with 48 procs per node. I re-run the case, using only 48 procs, thus there's no need to pass over a 'slow' interconnect. I'm now also using GAMG and BCGS for the poisson and momentum eqn respectively. I have also separated the x,y,z component of the momentum eqn to 3 separate linear eqns to debug the problem. Results show that stage "momentum_z" is taking a lot of time. I wonder if it has to do with the fact that I am partitioning my grids in the z direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. I wonder why a lot of entries are generated off-process. I create my RHS vector using: /call VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ where ijk_xyz_sta and ijk_xyz_end are obtained from /call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)/ I then insert the values into the vector using: /call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : ijk_xyz_end) , INSERT_VALUES , ierr)/ What should I do to correct the problem? Thanks > > Btw, I insert matrix using: > > /do ijk=ijk_xyz_sta+1,ijk_xyz_end// > // > // II = ijk - 1// !Fortran shift to 0-based// > //// > // call > MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// > // > //end do/ > > where ijk_xyz_sta/ijk_xyz_end are the starting/end index > > int_semi_xyz(ijk,1:7) stores the 7 column global indices > > semi_mat_xyz has the corresponding values. > > and I insert vectors using: > > call > VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) > > Thanks! > > / > / > > Yours sincerely, > > TAY wee-beng > > On 30/9/2012 11:30 PM, Jed Brown wrote: >> >> You can measure the time spent in Hypre via PCApply and PCSetUp, >> but you can't get finer grained integrated profiling because it >> was not set up that way. >> >> On Sep 30, 2012 3:26 PM, "TAY wee-beng" > > wrote: >> >> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng >>> > wrote: >>> >>> Hi, >>> >>> I'm doing a log summary for my 3d cfd code. I have some >>> questions: >>> >>> 1. if I'm solving 3 linear equations using ksp, is the >>> result given in the log summary the total of the 3 >>> linear eqns' performance? How can I get the performance >>> for each individual eqn? >>> >>> >>> Use logging stages: >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>> >>> 2. If I run my code for 10 time steps, does the log >>> summary gives the total or avg performance/ratio? >>> >>> >>> Total. >>> >>> 3. Besides PETSc, I'm also using HYPRE's native >>> geometric MG (Struct) to solve my Cartesian's grid CFD >>> poisson eqn. Is there any way I can use PETSc's log >>> summary to get HYPRE's performance? If I use boomerAMG >>> thru PETSc, can I get its performance? >>> >>> >>> If you mean flops, only if you count them yourself and tell >>> PETSc using >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>> >>> This is the disadvantage of using packages that do not >>> properly monitor things :) >>> >>> Matt >> So u mean if I use boomerAMG thru PETSc, there is no proper >> way of evaluating its performance, beside using PetscLogFlops? >>> >>> >>> -- >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-09 with 48 processors, by wtay Wed Oct 3 14:07:35 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 6.005e+03 1.00489 5.990e+03 Objects: 5.680e+02 1.00000 5.680e+02 Flops: 5.608e+11 1.24568 4.595e+11 2.205e+13 Flops/sec: 9.338e+07 1.24320 7.671e+07 3.682e+09 MPI Messages: 1.768e+05 4.44284 1.320e+05 6.337e+06 MPI Message Lengths: 1.524e+10 2.00538 1.128e+05 7.146e+11 MPI Reductions: 1.352e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.9025e+02 11.5% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 3.600e+01 0.3% 1: poisson: 1.5981e+02 2.7% 5.8320e+11 2.6% 1.749e+05 2.8% 3.211e+03 2.8% 1.435e+03 10.6% 2: momentum_x: 3.7257e+00 0.1% 7.1634e+09 0.0% 3.760e+02 0.0% 1.616e+01 0.0% 2.800e+01 0.2% 3: momentum_y: 4.5579e+00 0.1% 7.2165e+09 0.0% 3.760e+02 0.0% 1.638e+01 0.0% 2.800e+01 0.2% 4: momentum_z: 5.1317e+03 85.7% 2.1457e+13 97.3% 6.161e+06 97.2% 1.095e+05 97.1% 1.199e+04 88.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 949 1.0 8.0050e+01 1.4 8.30e+09 1.3 1.1e+05 1.3e+05 0.0e+00 1 1 2 2 0 42 52 65 73 0 3820 MatMultAdd 140 1.0 4.1770e+00 2.0 5.68e+08 1.5 1.3e+04 2.8e+04 0.0e+00 0 0 0 0 0 2 3 7 2 0 4421 MatMultTranspose 140 1.0 9.3398e+00 4.6 5.68e+08 1.5 1.3e+04 2.8e+04 0.0e+00 0 0 0 0 0 3 3 7 2 0 1977 MatSolve 70 0.0 1.3373e-03 0.0 7.18e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 537 MatLUFactorSym 1 1.0 2.8896e-0419.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 3.9911e-0450.7 1.89e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 473 MatConvert 4 1.0 3.9349e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 4.3059e-01 1.4 5.29e+07 1.4 5.0e+02 1.2e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 4320 MatAssemblyBegin 69 1.0 1.4660e+01 3.3 0.00e+00 0.0 1.3e+03 2.4e+04 7.4e+01 0 0 0 0 1 5 0 1 0 5 0 MatAssemblyEnd 69 1.0 3.5805e+00 1.1 0.00e+00 0.0 7.3e+03 1.7e+04 2.0e+02 0 0 0 0 1 2 0 4 1 14 0 MatGetRow 3004250 1.0 1.2984e+00 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 0.0 3.3140e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 1.2302e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 1.1238e+00 1.1 0.00e+00 0.0 1.5e+04 4.2e+04 3.4e+02 0 0 0 0 2 1 0 8 3 23 0 MatAXPY 4 1.0 1.4109e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 3.1390e+00 1.0 3.67e+07 1.4 3.3e+03 5.7e+04 9.6e+01 0 0 0 0 1 2 0 2 1 7 424 MatMatMultSym 4 1.0 2.3711e+00 1.1 0.00e+00 0.0 2.8e+03 4.6e+04 8.8e+01 0 0 0 0 1 1 0 2 1 6 0 MatMatMultNum 4 1.0 7.8353e-01 1.0 3.67e+07 1.4 5.0e+02 1.2e+05 8.0e+00 0 0 0 0 0 0 0 0 0 1 1701 MatPtAP 4 1.0 1.0608e+01 1.0 1.89e+09 3.5 6.5e+03 1.2e+05 1.1e+02 0 0 0 0 1 7 5 4 4 8 2808 MatPtAPSymbolic 4 1.0 5.7953e+00 1.0 0.00e+00 0.0 5.9e+03 9.6e+04 1.0e+02 0 0 0 0 1 4 0 3 3 7 0 MatPtAPNumeric 4 1.0 4.8300e+00 1.0 1.89e+09 3.5 6.4e+02 3.1e+05 8.0e+00 0 0 0 0 0 3 5 0 1 1 6167 MatTrnMatMult 4 1.0 2.3590e+01 1.0 1.56e+09 4.5 2.7e+03 8.7e+05 1.2e+02 0 0 0 0 1 15 12 2 11 8 3021 MatGetLocalMat 20 1.0 1.2136e+00 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 12 1.0 5.6065e-01 2.7 0.00e+00 0.0 3.5e+03 1.8e+05 1.6e+01 0 0 0 0 0 0 0 2 3 1 0 MatGetSymTrans 8 1.0 1.6375e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 75 1.0 2.5885e+00 2.0 3.30e+08 1.0 0.0e+00 0.0e+00 7.5e+01 0 0 0 0 1 1 3 0 0 5 5937 KSPSetUp 11 1.0 3.6090e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 KSPSolve 1 1.0 1.5654e+02 1.0 1.46e+10 1.3 1.7e+05 1.2e+05 1.4e+03 3 3 3 3 10 98100100100 99 3726 VecDot 34 1.0 7.1871e+00 6.1 9.03e+07 1.0 0.0e+00 0.0e+00 3.4e+01 0 0 0 0 0 3 1 0 0 2 602 VecDotNorm2 17 1.0 7.2364e+00 5.6 1.81e+08 1.0 0.0e+00 0.0e+00 5.1e+01 0 0 0 0 0 3 1 0 0 4 1196 VecMDot 75 1.0 2.2869e+00 3.8 1.65e+08 1.0 0.0e+00 0.0e+00 7.5e+01 0 0 0 0 1 1 1 0 0 5 3360 VecNorm 132 1.0 5.8965e+0011.2 8.09e+07 1.0 0.0e+00 0.0e+00 1.3e+02 0 0 0 0 1 2 1 0 0 9 649 VecScale 674 1.0 1.7632e+00 3.1 2.27e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 5983 VecCopy 181 1.0 6.9148e-01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 673 1.0 7.6137e-01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1159 1.0 4.6118e+00 3.0 8.44e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 8513 VecAYPX 1120 1.0 6.7999e+00 3.0 5.26e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 4 0 0 0 3596 VecAXPBYCZ 34 1.0 2.2253e+00 2.7 1.81e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 3890 VecWAXPY 34 1.0 2.3384e+00 3.0 9.03e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 1851 VecMAXPY 114 1.0 8.3695e-01 2.4 1.95e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 10851 VecAssemblyBegin 110 1.0 4.8131e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.2e+02 0 0 0 0 2 0 0 0 0 23 0 VecAssemblyEnd 110 1.0 3.8481e-04 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 884 1.0 7.6038e+00 3.3 3.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 3 0 0 0 2030 VecScatterBegin 1345 1.0 8.7802e-01 4.0 0.00e+00 0.0 1.6e+05 1.1e+05 0.0e+00 0 0 2 2 0 0 0 90 83 0 0 VecScatterEnd 1345 1.0 5.5070e+0113.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 16 0 0 0 0 0 VecSetRandom 4 1.0 1.0962e-01 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 114 1.0 4.4160e+0011.7 4.96e+07 1.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 1 1 0 0 0 8 522 PCSetUp 2 1.0 5.6255e+01 1.0 3.09e+09 1.2 4.1e+04 1.3e+05 1.2e+03 1 1 1 1 9 35 23 23 27 84 2433 PCSetUpOnBlocks 35 1.0 8.9216e-04 5.9 1.89e+05 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 212 PCApply 35 1.0 8.1400e+01 1.1 1.04e+10 1.3 1.3e+05 1.0e+05 1.2e+02 1 2 2 2 1 49 67 75 66 8 4792 PCGAMGgraph_AGG 4 1.0 6.8337e+00 1.0 3.67e+07 1.4 1.3e+03 7.0e+04 7.6e+01 0 0 0 0 1 4 0 1 0 5 195 PCGAMGcoarse_AGG 4 1.0 2.5930e+01 1.0 1.56e+09 4.5 2.0e+04 1.7e+05 5.3e+02 0 0 0 0 4 16 12 11 17 37 2748 PCGAMGProl_AGG 4 1.0 3.3635e+00 1.0 0.00e+00 0.0 3.4e+03 8.1e+04 1.1e+02 0 0 0 0 1 2 0 2 1 8 0 PCGAMGPOpt_AGG 4 1.0 9.5650e+00 1.0 8.23e+08 1.2 8.3e+03 9.6e+04 2.1e+02 0 0 0 0 2 6 6 5 4 15 3603 --- Event Stage 2: momentum_x MatMult 2 1.0 1.8518e-01 1.1 3.40e+07 1.1 1.9e+02 4.4e+05 0.0e+00 0 0 0 0 0 5 23 50 80 0 8786 MatSolve 3 1.0 2.5920e-01 1.2 5.03e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 34 0 0 0 9297 MatLUFactorNum 1 1.0 2.7748e-01 1.1 2.81e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 19 0 0 0 4845 MatILUFactorSym 1 1.0 2.8246e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 7 0 0 0 4 0 MatAssemblyBegin 1 1.0 5.8029e-013185.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 5 0 0 0 7 0 MatAssemblyEnd 1 1.0 4.6551e-01 1.1 0.00e+00 0.0 1.9e+02 1.1e+05 8.0e+00 0 0 0 0 0 12 0 50 20 29 0 MatGetRowIJ 1 1.0 1.3113e-05 6.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 3.4148e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 1.0453e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 1.2832e+00 1.0 1.50e+08 1.1 1.9e+02 4.4e+05 1.2e+01 0 0 0 0 0 34100 50 80 43 5583 VecDot 2 1.0 3.1625e-02 2.1 5.31e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 8051 VecDotNorm2 1 1.0 3.0667e-02 1.9 1.06e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 1 7 0 0 11 16605 VecNorm 2 1.0 8.6468e-0212.2 5.31e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 2945 VecCopy 2 1.0 2.2391e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 4.3424e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 5.0054e-02 1.5 1.06e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 10173 VecWAXPY 2 1.0 3.7987e-02 1.1 5.31e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 6703 VecAssemblyBegin 2 1.0 5.3350e-02206.6 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 1 0 0 0 21 0 VecAssemblyEnd 2 1.0 2.1935e-05 5.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 2.8200e-03 2.5 0.00e+00 0.0 1.9e+02 4.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 1.5276e-02 4.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 5.8300e-01 1.1 2.81e+07 1.1 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 15 19 0 0 18 2306 PCSetUpOnBlocks 1 1.0 5.8285e-01 1.1 2.81e+07 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 15 19 0 0 11 2307 PCApply 3 1.0 2.7368e-01 1.2 5.03e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 34 0 0 0 8805 --- Event Stage 3: momentum_y MatMult 2 1.0 2.6384e-01 1.6 3.43e+07 1.1 1.9e+02 4.4e+05 0.0e+00 0 0 0 0 0 5 23 50 80 0 6228 MatSolve 3 1.0 4.3936e-01 1.9 5.08e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 34 0 0 0 5539 MatLUFactorNum 1 1.0 3.7870e-01 1.3 2.83e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 19 0 0 0 3584 MatILUFactorSym 1 1.0 6.7441e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 10 0 0 0 4 0 MatAssemblyBegin 1 1.0 2.6880e-011700.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 3 0 0 0 7 0 MatAssemblyEnd 1 1.0 4.4579e-01 1.1 0.00e+00 0.0 1.9e+02 1.1e+05 8.0e+00 0 0 0 0 0 9 0 50 20 29 0 MatGetRowIJ 1 1.0 6.9141e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.4430e-01 6.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 1.9449e-01 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 2.1803e+00 1.0 1.51e+08 1.1 1.9e+02 4.4e+05 1.2e+01 0 0 0 0 0 48100 50 80 43 3310 VecDot 2 1.0 1.6268e-0112.1 5.31e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 2 4 0 0 7 1565 VecDotNorm2 1 1.0 1.5914e-01 4.8 1.06e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 3 7 0 0 11 3200 VecNorm 2 1.0 5.7848e-0184.8 5.31e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 8 4 0 0 7 440 VecCopy 2 1.0 4.2936e-02 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 8.5729e-02 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 9.1384e-02 2.7 1.06e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 5572 VecWAXPY 2 1.0 5.4740e-02 1.6 5.31e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 4651 VecAssemblyBegin 2 1.0 5.1133e-02181.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 1 0 0 0 21 0 VecAssemblyEnd 2 1.0 2.2888e-05 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 4.3221e-03 3.7 0.00e+00 0.0 1.9e+02 4.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 9.5706e-0248.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 2 1.0 1.1065e+00 1.7 2.83e+07 1.1 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 18 19 0 0 18 1227 PCSetUpOnBlocks 1 1.0 1.1063e+00 1.7 2.83e+07 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 18 19 0 0 11 1227 PCApply 3 1.0 4.5571e-01 1.8 5.08e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 34 0 0 0 5340 --- Event Stage 4: momentum_z MatMult 41867 1.0 3.5526e+03 1.3 3.68e+11 1.3 5.0e+06 1.3e+05 0.0e+00 51 62 79 93 0 60 64 81 95 0 3841 MatMultAdd 6404 1.0 2.5862e+02 2.7 2.60e+10 1.5 5.9e+05 2.8e+04 0.0e+00 2 4 9 2 0 3 4 10 2 0 3266 MatMultTranspose 6404 1.0 4.1748e+02 5.0 2.60e+10 1.5 5.9e+05 2.8e+04 0.0e+00 4 4 9 2 0 4 4 10 2 0 2023 MatSolve 3637 8.4 6.2673e+01 1.9 7.38e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 5614 MatLUFactorNum 145 1.0 6.3762e+01 1.6 4.10e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 3078 MatILUFactorSym 1 1.0 7.4252e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 145 1.0 4.6714e+01 8.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 0 0 0 0 2 0 0 0 0 2 0 MatAssemblyEnd 145 1.0 3.1650e+01 2.4 0.00e+00 0.0 1.9e+02 1.1e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 1.1921e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.4477e-01 4.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 1601 1.0 1.4767e-01 2.1 5.36e+05 0.0 0.0e+00 0.0e+00 1.6e+03 0 0 0 0 12 0 0 0 0 13 4 KSPSetUp 290 1.0 3.6730e-01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 194 1.0 4.8040e+03 1.0 5.46e+11 1.2 6.2e+06 1.1e+05 1.1e+04 80 97 97 97 78 94100100100 88 4466 VecDot 1842 1.0 3.9224e+02 8.1 4.89e+09 1.0 0.0e+00 0.0e+00 1.8e+03 3 1 0 0 14 4 1 0 0 15 598 VecDotNorm2 921 1.0 3.8957e+02 6.6 9.79e+09 1.0 0.0e+00 0.0e+00 2.8e+03 3 2 0 0 20 4 2 0 0 23 1204 VecMDot 1601 1.0 1.2703e-01 2.4 2.67e+05 0.0 0.0e+00 0.0e+00 1.6e+03 0 0 0 0 12 0 0 0 0 13 2 VecNorm 4317 1.0 2.5639e+0218.7 2.96e+09 1.0 0.0e+00 0.0e+00 4.3e+03 2 1 0 0 32 2 1 0 0 36 554 VecScale 28818 1.0 8.1507e+01 3.9 9.62e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 5489 VecCopy 8393 1.0 3.4342e+01 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 28235 1.0 3.8371e+01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 52833 1.0 2.2300e+02 3.3 3.85e+10 1.0 0.0e+00 0.0e+00 0.0e+00 2 8 0 0 0 3 8 0 0 0 8025 VecAYPX 51232 1.0 3.0984e+02 3.0 2.40e+10 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 4 5 0 0 0 3610 VecAXPBYCZ 1842 1.0 1.1255e+02 2.3 9.79e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 4167 VecWAXPY 1842 1.0 1.1471e+02 2.7 4.89e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2044 VecMAXPY 3202 1.0 1.0685e-02 5.7 5.38e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 50 VecAssemblyBegin 388 1.0 3.0027e+0160.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03 0 0 0 0 9 0 0 0 0 10 0 VecAssemblyEnd 388 1.0 3.5229e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 38424 1.0 3.4261e+02 3.4 1.44e+10 1.0 0.0e+00 0.0e+00 0.0e+00 3 3 0 0 0 4 3 0 0 0 1959 VecScatterBegin 54675 1.0 3.8022e+01 4.2 0.00e+00 0.0 6.2e+06 1.1e+05 0.0e+00 0 0 97 97 0 0 0100100 0 0 VecScatterEnd 54675 1.0 2.5222e+0320.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 23 0 0 0 0 0 VecNormalize 3202 1.0 1.6108e+021783.0 8.07e+05 0.0 0.0e+00 0.0e+00 3.2e+03 1 0 0 0 24 1 0 0 0 27 0 PCSetUp 290 1.0 6.4608e+01 1.6 4.10e+09 1.1 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 1 1 0 0 0 3038 PCSetUpOnBlocks 1746 1.0 6.4608e+01 1.6 4.10e+09 1.1 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 1 1 0 0 0 3038 PCApply 2036 1.0 3.8864e+03 1.2 4.81e+11 1.3 6.0e+06 1.0e+05 4.8e+03 61 83 94 86 36 71 85 97 89 40 4682 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 12 56 1787149752 0 Krylov Solver 4 13 31728 0 Vector 8 85 405124904 0 Vector Scatter 0 12 12720 0 Index Set 0 16 15953504 0 Preconditioner 0 13 12540 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 117 76 1166561340 0 Matrix Coarsen 4 4 2480 0 Krylov Solver 10 4 120512 0 Vector 225 175 331789264 0 Vector Scatter 31 22 23320 0 Index Set 81 74 1327712 0 Preconditioner 11 4 3456 0 PetscRandom 4 4 2464 0 --- Event Stage 2: momentum_x Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 219352 0 Preconditioner 2 0 0 0 --- Event Stage 3: momentum_y Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 222304 0 Preconditioner 2 0 0 0 --- Event Stage 4: momentum_z Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 222296 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 Average time for MPI_Barrier(): 3.71933e-05 Average time for zero size MPI_Send(): 1.50005e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Mon Oct 1 11:36:09 2012 on hpc12 Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.3-dev_shared_rel ----------------------------------------- From gdiso at ustc.edu Wed Oct 3 10:00:56 2012 From: gdiso at ustc.edu (Gong Ding) Date: Wed, 3 Oct 2012 23:00:56 +0800 (CST) Subject: [petsc-users] More about SNESSetLagPreconditioner Message-ID: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> Hi, I am doing a nonlinear transient simulation. For each time step, a SNESSolve is called. I want to do a LU factorization at the first Newton iteration of EACH time step, and for the following Newton steps (in the same time step), no factorization any more. For the next time step, the LU factorization should be done again. SNESSetLagPreconditioner(-1) seems just skips all the following factorization, even in the different time step. How can I do? Gong Ding From jedbrown at mcs.anl.gov Wed Oct 3 10:06:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Oct 2012 11:06:57 -0400 Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> Message-ID: I've been meaning to implement this for a while. You can put custom SNES control into a pre-stage or pre-step hook, but this particular case should, in my opinion, be provided as a TS option. On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: > Hi, > I am doing a nonlinear transient simulation. For each time step, a > SNESSolve is called. > I want to do a LU factorization at the first Newton iteration of EACH time > step, and for the following Newton steps (in the same time step), no > factorization any more. For the next time step, the LU factorization should > be done again. > > > SNESSetLagPreconditioner(-1) seems just skips all the following > factorization, even in the different time step. > How can I do? > > > Gong Ding > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 3 10:59:59 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Oct 2012 10:59:59 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506C38DE.7030107@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> Message-ID: There is an inordinate amount of time being spent in VecScatterEnd(). That sometimes indicates a very bad partition. Also, are your "48 cores" real physical cores or just "logical cores" (look like cores to the operating system, usually advertised as "threads" by the vendor, nothing like cores in reality)? That can cause a huge load imbalance and very confusing results as over-subscribed threads compete for shared resources. Step it back to 24 threads and 12 threads, send log_summary for each. On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: > On 2/10/2012 2:43 PM, Jed Brown wrote: > > On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: > >> Hi, >> >> I have combined the momentum linear eqns involving x,y,z into 1 large >> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >> procs. The log_summary is given below. I have some questions: >> >> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >> does it says there are 4 matrix, 12 vector etc? >> > > They are part of preconditioning. Are you sure you're using Hypre for > this? It looks like you are using bjacobi/ilu. > > >> >> 2. I'm looking at the stages which take the longest time. It seems that >> MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high >> ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are >> these stages the reason why my code is not scaling well? What can I do to >> improve it? >> > > 3/4 of the solve time is evenly balanced between MatMult, MatSolve, > MatLUFactorNumeric, and VecNorm+VecDot. > > The high VecAssembly time might be due to generating a lot of entries > off-process? > > In any case, this looks like an _extremely_ slow network, perhaps it's > misconfigured? > > > My cluster is configured with 48 procs per node. I re-run the case, using > only 48 procs, thus there's no need to pass over a 'slow' interconnect. I'm > now also using GAMG and BCGS for the poisson and momentum eqn respectively. > I have also separated the x,y,z component of the momentum eqn to 3 separate > linear eqns to debug the problem. > > Results show that stage "momentum_z" is taking a lot of time. I wonder if > it has to do with the fact that I am partitioning my grids in the z > direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, > VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. > > I wonder why a lot of entries are generated off-process. > > I create my RHS vector using: > > *call > VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) > * > > where ijk_xyz_sta and ijk_xyz_end are obtained from > > *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* > > I then insert the values into the vector using: > > *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , > (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : > ijk_xyz_end) , INSERT_VALUES , ierr)* > > What should I do to correct the problem? > > Thanks > > > > >> >> Btw, I insert matrix using: >> >> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >> ** >> ** II = ijk - 1** !Fortran shift to 0-based** >> ** ** >> ** call >> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >> ** >> ** >> **end do* >> >> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >> >> int_semi_xyz(ijk,1:7) stores the 7 column global indices >> >> semi_mat_xyz has the corresponding values. >> >> and I insert vectors using: >> >> call >> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >> >> Thanks! >> >> * >> * >> >> Yours sincerely, >> >> TAY wee-beng >> >> On 30/9/2012 11:30 PM, Jed Brown wrote: >> >> You can measure the time spent in Hypre via PCApply and PCSetUp, but you >> can't get finer grained integrated profiling because it was not set up that >> way. >> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >> >>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>> >>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>> >>>> Hi, >>>> >>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>> >>>> 1. if I'm solving 3 linear equations using ksp, is the result given in >>>> the log summary the total of the 3 linear eqns' performance? How can I get >>>> the performance for each individual eqn? >>>> >>> >>> Use logging stages: >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>> >>> >>>> 2. If I run my code for 10 time steps, does the log summary gives the >>>> total or avg performance/ratio? >>>> >>> >>> Total. >>> >>> >>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) >>>> to solve my Cartesian's grid CFD poisson eqn. Is there any way I can use >>>> PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru >>>> PETSc, can I get its performance? >>> >>> >>> If you mean flops, only if you count them yourself and tell PETSc >>> using >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>> >>> This is the disadvantage of using packages that do not properly >>> monitor things :) >>> >>> Matt >>> >>> >>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>> evaluating its performance, beside using PetscLogFlops? >>> >>> >>>> -- >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siaeleni at hotmail.com Wed Oct 3 11:58:53 2012 From: siaeleni at hotmail.com (Eleni Siampli) Date: Wed, 3 Oct 2012 19:58:53 +0300 Subject: [petsc-users] PetscFinalize() In-Reply-To: References: , , , Message-ID: Hello again, It doen't work good. I have created a dll for solving the eigenPorblem (eigenSolver.dll) and I included it in my current project. lets say that eigenSolver is the name of my class.. What I use is: eigensolver eig; for a lot of iterations and works fine. I call only one the PetscInitialize(); I want also to use: eigensolver eig1; eigensolver eig2; I call only once PetscInitialize(); for each eig1 and eig 2. I have the same error as before. Why does it give me the same error?I call only once the PetscInitialize(). Thanks a lot in advance, Helen Date: Fri, 28 Sep 2012 20:23:20 -0400 From: knepley at gmail.com To: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] PetscFinalize() On Fri, Sep 28, 2012 at 5:09 PM, Eleni Siampli wrote: Thank you for the answer. Problem solved, but now I have another one: After a lot of iterations (around 246) It gives me the following error: [0]PETSC ERROR: Petsc has generated inconsistent data! [0]PETSC ERROR: No more room in array, limit 256 recompile src/sys/objects/destroy.c with larger value for MAXREGFIN Do you have any idea about this problem? You are also calling PetscInitialize() a bunch of times. Its intended for just one. Matt Thanks, Helen Date: Fri, 28 Sep 2012 11:17:51 -0400 From: knepley at gmail.com To: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] PetscFinalize() On Fri, Sep 28, 2012 at 11:16 AM, Eleni Siampli wrote: Hey all, I would like to solve the eigenvalue problem. My code is working for 2 loops and give the right values, but after two loops it gives me the following error: [0]PETSC ERROR: PetscFinalize() line 1221 in src/sys/objects/C:\cygwin\home\liyi 0000\PETSC-~1.2-P\src\sys\objects\pinit.c [0]PETSC ERROR: SlepcFinalize() line 224 in src/sys/C:\cygwin\home\liyi0000\SLEP C-~1.2-P\src\sys\slepcinit.c Options have not been enabled. You might have forgotten to call PetscInitialize(). Do you have any idea what this error mean? You are only intended to call PetscFinalize() once, at the end of your program. Matt Thank you in advance, Helen -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Oct 3 14:16:07 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Oct 2012 14:16:07 -0500 Subject: [petsc-users] PetscFinalize() In-Reply-To: References: , , , Message-ID: <472BDE33-C313-4BEF-83B1-ED75CB2D273B@mcs.anl.gov> This error occurs if more than 250 functions are registered with PetscRegisterFinalize(). Under normal usage that should never happen. Likely something is calling PetscRegisterFinalize() unprotected, that is, calling it many times with the same function to register. Likely this is in SLEPc. The other possibility is some strange interaction with .dll on Microsoft Windows? First I would start your program in the debugger, put a break point in PetscRegisterFinalize() and then run the program. Each time it breaks in PetscRegisterFinalize() note where it is called from. Under normal circumstances PETSc should call PetscRegisterFinalize() around 10 times. If you see it called 30 times note what functions are calling it over and over again and send back that information. Barry On Oct 3, 2012, at 11:58 AM, Eleni Siampli wrote: > > Hello again, > > It doen't work good. I have created a dll for solving the eigenPorblem (eigenSolver.dll) and I included it in my current project. > lets say that eigenSolver is the name of my class.. What I use is: > eigensolver eig; > for a lot of iterations and works fine. I call only one the PetscInitialize(); > > I want also to use: > eigensolver eig1; > eigensolver eig2; > > I call only once PetscInitialize(); for each eig1 and eig 2. > > I have the same error as before. > > Why does it give me the same error?I call only once the PetscInitialize(). > Thanks a lot in advance, > Helen > > Date: Fri, 28 Sep 2012 20:23:20 -0400 > From: knepley at gmail.com > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PetscFinalize() > > On Fri, Sep 28, 2012 at 5:09 PM, Eleni Siampli wrote: > > Thank you for the answer. Problem solved, but now I have another one: > > After a lot of iterations (around 246) It gives me the following error: > > > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: No more room in array, limit 256 > recompile src/sys/objects/destroy.c with larger value for MAXREGFIN > > Do you have any idea about this problem? > > You are also calling PetscInitialize() a bunch of times. Its intended for just one. > > Matt > > > Thanks, > Helen > Date: Fri, 28 Sep 2012 11:17:51 -0400 > From: knepley at gmail.com > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PetscFinalize() > > On Fri, Sep 28, 2012 at 11:16 AM, Eleni Siampli wrote: > Hey all, > > I would like to solve the eigenvalue problem. > My code is working for 2 loops and give the right values, but after two loops it gives me the following error: > > [0]PETSC ERROR: PetscFinalize() line 1221 in src/sys/objects/C:\cygwin\home\liyi > 0000\PETSC-~1.2-P\src\sys\objects\pinit.c > [0]PETSC ERROR: SlepcFinalize() line 224 in src/sys/C:\cygwin\home\liyi0000\SLEP > C-~1.2-P\src\sys\slepcinit.c > Options have not been enabled. > You might have forgotten to call PetscInitialize(). > > Do you have any idea what this error mean? > > You are only intended to call PetscFinalize() once, at the end of your program. > > Matt > > > Thank you in advance, > Helen > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener From zonexo at gmail.com Wed Oct 3 15:05:45 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Wed, 03 Oct 2012 22:05:45 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> Message-ID: <506C9A99.60300@gmail.com> Hi Jed, I believe they are real cores. Anyway, I have attached the log summary for the 12/24/48 cores. I re-run a smaller case because the large problem can't run with 12cores. Yours sincerely, TAY wee-beng On 3/10/2012 5:59 PM, Jed Brown wrote: > There is an inordinate amount of time being spent in VecScatterEnd(). > That sometimes indicates a very bad partition. Also, are your "48 > cores" real physical cores or just "logical cores" (look like cores to > the operating system, usually advertised as "threads" by the vendor, > nothing like cores in reality)? That can cause a huge load imbalance > and very confusing results as over-subscribed threads compete for > shared resources. Step it back to 24 threads and 12 threads, send > log_summary for each. > > On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng > wrote: > > On 2/10/2012 2:43 PM, Jed Brown wrote: >> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng > > wrote: >> >> Hi, >> >> I have combined the momentum linear eqns involving x,y,z into >> 1 large matrix. The Poisson eqn is solved using HYPRE strcut >> format so it's not included. I run the code for 50 timesteps >> (hence 50 kspsolve) using 96 procs. The log_summary is given >> below. I have some questions: >> >> 1. After combining the matrix, I should have only 1 PETSc >> matrix. Why does it says there are 4 matrix, 12 vector etc? >> >> >> They are part of preconditioning. Are you sure you're using Hypre >> for this? It looks like you are using bjacobi/ilu. >> >> >> 2. I'm looking at the stages which take the longest time. It >> seems that MatAssemblyBegin, VecNorm, VecAssemblyBegin, >> VecScatterEnd have very high ratios. The ratios of some >> others are also not too good (~ 1.6 - 2). So are these stages >> the reason why my code is not scaling well? What can I do to >> improve it? >> >> >> 3/4 of the solve time is evenly balanced between MatMult, >> MatSolve, MatLUFactorNumeric, and VecNorm+VecDot. >> >> The high VecAssembly time might be due to generating a lot of >> entries off-process? >> >> In any case, this looks like an _extremely_ slow network, perhaps >> it's misconfigured? > > My cluster is configured with 48 procs per node. I re-run the > case, using only 48 procs, thus there's no need to pass over a > 'slow' interconnect. I'm now also using GAMG and BCGS for the > poisson and momentum eqn respectively. I have also separated the > x,y,z component of the momentum eqn to 3 separate linear eqns to > debug the problem. > > Results show that stage "momentum_z" is taking a lot of time. I > wonder if it has to do with the fact that I am partitioning my > grids in the z direction. VecScatterEnd, MatMult are taking a lot > of time. VecNormalize, VecScatterEnd, VecNorm, VecAssemblyBegin 's > ratio are also not good. > > I wonder why a lot of entries are generated off-process. > > I create my RHS vector using: > > /call > VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ > > where ijk_xyz_sta and ijk_xyz_end are obtained from > > /call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)/ > > I then insert the values into the vector using: > > /call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , > (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 > : ijk_xyz_end) , INSERT_VALUES , ierr)/ > > What should I do to correct the problem? > > Thanks > > >> >> Btw, I insert matrix using: >> >> /do ijk=ijk_xyz_sta+1,ijk_xyz_end// >> // >> // II = ijk - 1// !Fortran shift to 0-based// >> //// >> // call >> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// >> // >> //end do/ >> >> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >> >> int_semi_xyz(ijk,1:7) stores the 7 column global indices >> >> semi_mat_xyz has the corresponding values. >> >> and I insert vectors using: >> >> call >> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >> >> Thanks! >> >> / >> / >> >> Yours sincerely, >> >> TAY wee-beng >> >> On 30/9/2012 11:30 PM, Jed Brown wrote: >>> >>> You can measure the time spent in Hypre via PCApply and >>> PCSetUp, but you can't get finer grained integrated >>> profiling because it was not set up that way. >>> >>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" >> > wrote: >>> >>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng >>>> > wrote: >>>> >>>> Hi, >>>> >>>> I'm doing a log summary for my 3d cfd code. I have >>>> some questions: >>>> >>>> 1. if I'm solving 3 linear equations using ksp, is >>>> the result given in the log summary the total of >>>> the 3 linear eqns' performance? How can I get the >>>> performance for each individual eqn? >>>> >>>> >>>> Use logging stages: >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>> >>>> 2. If I run my code for 10 time steps, does the log >>>> summary gives the total or avg performance/ratio? >>>> >>>> >>>> Total. >>>> >>>> 3. Besides PETSc, I'm also using HYPRE's native >>>> geometric MG (Struct) to solve my Cartesian's grid >>>> CFD poisson eqn. Is there any way I can use PETSc's >>>> log summary to get HYPRE's performance? If I use >>>> boomerAMG thru PETSc, can I get its performance? >>>> >>>> >>>> If you mean flops, only if you count them yourself and >>>> tell PETSc using >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>> >>>> This is the disadvantage of using packages that do not >>>> properly monitor things :) >>>> >>>> Matt >>> So u mean if I use boomerAMG thru PETSc, there is no >>> proper way of evaluating its performance, beside using >>> PetscLogFlops? >>>> >>>> >>>> -- >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they >>>> begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-10 with 12 processors, by wtay Wed Oct 3 21:34:22 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 4.407e+03 1.00084 4.405e+03 Objects: 5.680e+02 1.00000 5.680e+02 Flops: 7.375e+11 1.11505 6.777e+11 8.133e+12 Flops/sec: 1.674e+08 1.11490 1.539e+08 1.846e+09 MPI Messages: 1.460e+05 2.95457 1.080e+05 1.296e+06 MPI Message Lengths: 1.035e+10 2.02602 8.761e+04 1.136e+11 MPI Reductions: 1.571e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.6484e+02 6.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 3.600e+01 0.2% 1: poisson: 1.1297e+02 2.6% 1.9050e+11 2.3% 2.900e+04 2.2% 2.173e+03 2.5% 1.215e+03 7.7% 2: momentum_x: 2.8093e+00 0.1% 2.1513e+09 0.0% 8.800e+01 0.0% 1.010e+01 0.0% 2.800e+01 0.2% 3: momentum_y: 2.8534e+00 0.1% 2.1666e+09 0.0% 8.800e+01 0.0% 1.023e+01 0.0% 2.800e+01 0.2% 4: momentum_z: 4.0213e+03 91.3% 7.9380e+12 97.6% 1.267e+06 97.7% 8.541e+04 97.5% 1.440e+04 91.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 1053 1.0 5.4872e+01 1.3 9.42e+09 1.2 2.0e+04 1.1e+05 0.0e+00 1 1 2 2 0 44 53 69 75 0 1843 MatMultAdd 156 1.0 4.5059e+00 1.3 6.01e+08 1.2 2.7e+03 1.9e+04 0.0e+00 0 0 0 0 0 4 3 9 2 0 1358 MatMultTranspose 156 1.0 5.6541e+00 1.6 6.01e+08 1.2 2.7e+03 1.9e+04 0.0e+00 0 0 0 0 0 4 3 9 2 0 1082 MatSolve 78 0.0 4.3106e-04 0.0 6.75e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 157 MatLUFactorSym 1 1.0 4.1008e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 3.5048e-05 2.9 5.53e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 158 MatConvert 4 1.0 1.9400e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.9127e-01 1.2 5.26e+07 1.2 7.8e+01 1.0e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 2896 MatAssemblyBegin 69 1.0 3.9459e+00 2.0 0.00e+00 0.0 2.3e+02 1.8e+04 7.4e+01 0 0 0 0 0 2 0 1 0 6 0 MatAssemblyEnd 69 1.0 1.9654e+00 1.1 0.00e+00 0.0 1.0e+03 1.6e+04 2.0e+02 0 0 0 0 1 2 0 4 1 16 0 MatGetRow 3544348 1.0 8.9238e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 0.0 1.6928e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 7.7009e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.7e-01 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 6.2370e-01 1.1 0.00e+00 0.0 4.3e+02 1.4e+05 9.2e+01 0 0 0 0 1 1 0 1 2 8 0 MatAXPY 4 1.0 6.2184e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 1.7033e+00 1.0 3.72e+07 1.2 4.9e+02 4.9e+04 9.6e+01 0 0 0 0 1 2 0 2 1 8 233 MatMatMultSym 4 1.0 1.2570e+00 1.0 0.00e+00 0.0 4.1e+02 3.9e+04 8.8e+01 0 0 0 0 1 1 0 1 1 7 0 MatMatMultNum 4 1.0 4.5279e-01 1.0 3.72e+07 1.2 7.8e+01 1.0e+05 8.0e+00 0 0 0 0 0 0 0 0 0 1 877 MatPtAP 4 1.0 6.5757e+00 1.0 1.16e+09 1.8 8.4e+02 1.1e+05 1.1e+02 0 0 0 0 1 6 5 3 3 9 1304 MatPtAPSymbolic 4 1.0 3.6004e+00 1.0 0.00e+00 0.0 7.5e+02 9.3e+04 1.0e+02 0 0 0 0 1 3 0 3 2 8 0 MatPtAPNumeric 4 1.0 2.9819e+00 1.0 1.16e+09 1.8 8.9e+01 2.7e+05 8.0e+00 0 0 0 0 0 3 5 0 1 1 2876 MatTrnMatMult 4 1.0 1.6466e+01 1.0 2.04e+09 1.9 4.7e+02 6.6e+05 1.2e+02 0 0 0 0 1 15 12 2 11 10 1385 MatGetLocalMat 20 1.0 6.5459e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 12 1.0 1.7862e-01 2.1 0.00e+00 0.0 5.5e+02 1.4e+05 1.6e+01 0 0 0 0 0 0 0 2 3 1 0 MatGetSymTrans 8 1.0 8.3050e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 79 1.0 8.9525e-01 1.7 3.90e+08 1.0 0.0e+00 0.0e+00 7.9e+01 0 0 0 0 1 1 2 0 0 7 5139 KSPSetUp 11 1.0 1.8398e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 KSPSolve 1 1.0 1.1039e+02 1.0 1.67e+10 1.1 2.9e+04 9.7e+04 1.2e+03 3 2 2 2 8 98100100100 99 1726 VecDot 38 1.0 3.4189e+00 2.9 1.22e+08 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 0 2 1 0 0 3 424 VecDotNorm2 19 1.0 3.2140e+00 2.7 2.44e+08 1.0 0.0e+00 0.0e+00 5.7e+01 0 0 0 0 0 2 2 0 0 5 901 VecMDot 79 1.0 5.6992e-01 2.2 1.95e+08 1.0 0.0e+00 0.0e+00 7.9e+01 0 0 0 0 1 0 1 0 0 7 4037 VecNorm 142 1.0 2.7666e+00 5.7 1.03e+08 1.0 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 1 1 1 0 0 12 442 VecScale 746 1.0 1.2098e+00 2.0 2.96e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 2887 VecCopy 201 1.0 4.0863e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 741 1.0 3.8170e-01 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1291 1.0 4.7930e+00 2.7 1.11e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 2732 VecAYPX 1248 1.0 4.8211e+00 2.0 6.91e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 4 0 0 0 1692 VecAXPBYCZ 38 1.0 9.7988e-01 2.0 2.44e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 2956 VecWAXPY 38 1.0 9.6000e-01 2.5 1.22e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 1509 VecMAXPY 122 1.0 3.9971e-01 1.5 2.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 6802 VecAssemblyBegin 49 1.0 1.4084e-01 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 1 0 0 0 0 12 0 VecAssemblyEnd 49 1.0 1.4496e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 980 1.0 7.1430e+00 3.3 4.34e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 3 0 0 0 717 VecScatterBegin 1420 1.0 2.7552e-01 3.7 0.00e+00 0.0 2.6e+04 9.0e+04 0.0e+00 0 0 2 2 0 0 0 91 84 0 0 VecScatterEnd 1420 1.0 3.7127e+0112.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 VecSetRandom 4 1.0 4.9694e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 122 1.0 2.3469e+0021.0 5.85e+07 1.0 0.0e+00 0.0e+00 1.2e+02 0 0 0 0 1 1 0 0 0 10 294 PCSetUp 2 1.0 3.5545e+01 1.0 3.60e+09 1.1 4.4e+03 1.6e+05 9.6e+02 1 1 0 1 6 31 22 15 24 79 1184 PCSetUpOnBlocks 39 1.0 2.9898e-04 1.7 5.53e+03 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 18 PCApply 39 1.0 6.6028e+01 1.1 1.20e+10 1.1 2.4e+04 8.1e+04 1.3e+02 1 2 2 2 1 56 68 82 68 11 1963 PCGAMGgraph_AGG 4 1.0 4.2616e+00 1.0 3.72e+07 1.2 2.3e+02 5.1e+04 7.6e+01 0 0 0 0 0 4 0 1 0 6 93 PCGAMGcoarse_AGG 4 1.0 1.7933e+01 1.0 2.04e+09 1.9 1.3e+03 3.4e+05 2.9e+02 0 0 0 0 2 16 12 5 16 24 1271 PCGAMGProl_AGG 4 1.0 2.2483e+00 1.0 0.00e+00 0.0 5.5e+02 6.4e+04 1.1e+02 0 0 0 0 1 2 0 2 1 9 0 PCGAMGPOpt_AGG 4 1.0 4.4578e+00 1.0 9.11e+08 1.1 1.3e+03 8.0e+04 2.1e+02 0 0 0 0 1 4 5 4 4 17 2310 --- Event Stage 2: momentum_x MatMult 2 1.0 9.9061e-02 1.1 4.10e+07 1.0 4.4e+01 2.4e+05 0.0e+00 0 0 0 0 0 3 23 50 80 0 4907 MatSolve 3 1.0 1.4430e-01 1.1 6.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 5026 MatLUFactorNum 1 1.0 2.5046e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 19 0 0 0 1622 MatILUFactorSym 1 1.0 1.5358e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 5 0 0 0 4 0 MatAssemblyBegin 1 1.0 2.8467e-013030.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 4 0 0 0 7 0 MatAssemblyEnd 1 1.0 2.2157e-01 1.1 0.00e+00 0.0 4.4e+01 5.9e+04 8.0e+00 0 0 0 0 0 8 0 50 20 29 0 MatGetRowIJ 1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.8516e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 6.5584e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 8.0632e-01 1.0 1.81e+08 1.0 4.4e+01 2.4e+05 1.2e+01 0 0 0 0 0 29100 50 80 43 2668 VecDot 2 1.0 1.6051e-02 2.0 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 4 0 0 7 4750 VecDotNorm2 1 1.0 1.5273e-02 1.7 1.28e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 7 0 0 11 9983 VecNorm 2 1.0 6.1728e-0212.2 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 1235 VecCopy 2 1.0 1.0178e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.7939e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 1.9800e-02 1.4 1.28e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 7700 VecWAXPY 2 1.0 1.6218e-02 1.4 6.42e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 4701 VecAssemblyBegin 2 1.0 2.2810e-02154.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 21 0 VecAssemblyEnd 2 1.0 9.0599e-06 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 7.9203e-04 2.3 0.00e+00 0.0 4.4e+01 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 1.6581e-0214.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 4.2014e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 14 19 0 0 18 967 PCSetUpOnBlocks 1 1.0 4.1998e-01 1.1 3.43e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 14 19 0 0 11 968 PCApply 3 1.0 1.4989e-01 1.1 6.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 4838 --- Event Stage 3: momentum_y MatMult 2 1.0 1.0025e-01 1.1 4.14e+07 1.0 4.4e+01 2.4e+05 0.0e+00 0 0 0 0 0 3 23 50 80 0 4896 MatSolve 3 1.0 1.4338e-01 1.1 6.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 5107 MatLUFactorNum 1 1.0 2.6953e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 19 0 0 0 1521 MatILUFactorSym 1 1.0 1.5543e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 5 0 0 0 4 0 MatAssemblyBegin 1 1.0 3.4289e-013174.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 5 0 0 0 7 0 MatAssemblyEnd 1 1.0 2.0907e-01 1.1 0.00e+00 0.0 4.4e+01 6.0e+04 8.0e+00 0 0 0 0 0 7 0 50 20 29 0 MatGetRowIJ 1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.8706e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 6.5596e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 8.3469e-01 1.0 1.83e+08 1.0 4.4e+01 2.4e+05 1.2e+01 0 0 0 0 0 29100 50 80 43 2596 VecDot 2 1.0 1.5645e-02 1.9 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 4 0 0 7 4873 VecDotNorm2 1 1.0 1.5150e-02 1.7 1.28e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 7 0 0 11 10064 VecNorm 2 1.0 7.3364e-02 6.5 6.42e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 1039 VecCopy 2 1.0 1.0147e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.7555e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 2.6077e-02 1.9 1.28e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 5847 VecWAXPY 2 1.0 1.6632e-02 1.5 6.42e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 4584 VecAssemblyBegin 2 1.0 1.9942e-02134.7 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 21 0 VecAssemblyEnd 2 1.0 8.1062e-06 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 7.4911e-04 2.2 0.00e+00 0.0 4.4e+01 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 1.8597e-0215.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 4.4319e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 15 19 0 0 18 925 PCSetUpOnBlocks 1 1.0 4.4305e-01 1.1 3.46e+07 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 15 19 0 0 11 925 PCApply 3 1.0 1.4899e-01 1.1 6.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 4914 --- Event Stage 4: momentum_z MatMult 52319 1.0 2.3522e+03 1.1 4.72e+11 1.1 9.9e+05 1.1e+05 0.0e+00 52 62 76 93 0 57 64 78 95 0 2157 MatMultAdd 8012 1.0 1.6520e+02 1.2 3.09e+10 1.2 1.4e+05 1.9e+04 0.0e+00 4 4 11 2 0 4 4 11 2 0 1902 MatMultTranspose 8012 1.0 2.4016e+02 1.4 3.09e+10 1.2 1.4e+05 1.9e+04 0.0e+00 5 4 11 2 0 5 4 11 2 0 1309 MatSolve 444110.2 4.4971e+01 1.7 8.93e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2354 MatLUFactorNum 145 1.0 4.3540e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1362 MatILUFactorSym 1 1.0 1.5319e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 145 1.0 3.5545e+01 8.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 1 0 0 0 2 1 0 0 0 2 0 MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 9.5367e-07 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.8826e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 2003 1.0 5.7039e-02 1.3 1.74e+05 0.0 0.0e+00 0.0e+00 2.0e+03 0 0 0 0 13 0 0 0 0 14 3 KSPSetUp 290 1.0 6.3499e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 194 1.0 3.7263e+03 1.0 7.20e+11 1.1 1.3e+06 8.7e+04 1.3e+04 85 98 98 97 82 93100100100 90 2130 VecDot 2244 1.0 1.6008e+02 1.9 7.21e+09 1.0 0.0e+00 0.0e+00 2.2e+03 3 1 0 0 14 3 1 0 0 16 534 VecDotNorm2 1122 1.0 1.5508e+02 1.8 1.44e+10 1.0 0.0e+00 0.0e+00 3.4e+03 3 2 0 0 21 3 2 0 0 23 1103 VecMDot 2003 1.0 4.2973e-02 1.4 8.61e+04 0.0 0.0e+00 0.0e+00 2.0e+03 0 0 0 0 13 0 0 0 0 14 2 VecNorm 5322 1.0 1.1301e+02 3.1 4.23e+09 1.0 0.0e+00 0.0e+00 5.3e+03 1 1 0 0 34 1 1 0 0 37 444 VecScale 36054 1.0 5.6512e+01 1.3 1.42e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 2965 VecCopy 10403 1.0 2.1793e+01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 35069 1.0 1.9805e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 66099 1.0 1.7941e+02 1.3 5.68e+10 1.0 0.0e+00 0.0e+00 0.0e+00 4 8 0 0 0 4 8 0 0 0 3736 VecAYPX 64096 1.0 1.9224e+02 1.3 3.55e+10 1.0 0.0e+00 0.0e+00 0.0e+00 4 5 0 0 0 4 5 0 0 0 2179 VecAXPBYCZ 2244 1.0 5.0866e+01 1.5 1.44e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 3363 VecWAXPY 2244 1.0 4.8066e+01 1.4 7.21e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1780 VecMAXPY 4006 1.0 5.5676e-03 1.5 1.76e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 32 VecAssemblyBegin 388 1.0 2.0726e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03 0 0 0 0 7 0 0 0 0 8 0 VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 48072 1.0 2.3956e+02 1.6 2.13e+10 1.0 0.0e+00 0.0e+00 0.0e+00 5 3 0 0 0 5 3 0 0 0 1049 VecScatterBegin 68343 1.0 1.0083e+01 2.4 0.00e+00 0.0 1.3e+06 8.7e+04 0.0e+00 0 0 98 97 0 0 0100100 0 0 VecScatterEnd 68343 1.0 1.1153e+03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 19 0 0 0 0 20 0 0 0 0 0 VecNormalize 4006 1.0 5.8749e+01698.9 2.64e+05 0.0 0.0e+00 0.0e+00 4.0e+03 0 0 0 0 25 0 0 0 0 28 0 PCSetUp 290 1.0 4.3717e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 1 1 0 0 0 1357 PCSetUpOnBlocks 2148 1.0 4.3718e+01 1.2 5.00e+09 1.0 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 1 1 0 0 0 1357 PCApply 2438 1.0 3.1165e+03 1.0 6.24e+11 1.1 1.2e+06 8.1e+04 6.0e+03 70 83 94 87 38 76 85 96 89 42 2170 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 12 56 2093884688 0 Krylov Solver 4 13 31728 0 Vector 8 85 485916024 0 Vector Scatter 0 12 12720 0 Index Set 0 16 19283008 0 Preconditioner 0 13 12540 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 117 76 1488820988 0 Matrix Coarsen 4 4 2480 0 Krylov Solver 10 4 120512 0 Vector 225 175 387307616 0 Vector Scatter 31 22 23320 0 Index Set 81 74 808508 0 Preconditioner 11 4 3456 0 PetscRandom 4 4 2464 0 --- Event Stage 2: momentum_x Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 120488 0 Preconditioner 2 0 0 0 --- Event Stage 3: momentum_y Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122104 0 Preconditioner 2 0 0 0 --- Event Stage 4: momentum_z Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122096 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 1.03951e-05 Average time for zero size MPI_Send(): 1.1007e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Mon Oct 1 11:36:09 2012 on hpc12 Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.3-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-09 with 24 processors, by wtay Wed Oct 3 21:19:36 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 3.060e+03 1.00172 3.057e+03 Objects: 5.680e+02 1.00000 5.680e+02 Flops: 3.503e+11 1.23913 2.909e+11 6.982e+12 Flops/sec: 1.145e+08 1.23763 9.515e+07 2.284e+09 MPI Messages: 1.542e+05 3.65607 9.905e+04 2.377e+06 MPI Message Lengths: 8.847e+09 2.01744 8.530e+04 2.028e+11 MPI Reductions: 1.398e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.1590e+02 7.1% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 3.600e+01 0.3% 1: poisson: 6.7753e+01 2.2% 1.9064e+11 2.7% 6.527e+04 2.7% 2.477e+03 2.9% 1.307e+03 9.4% 2: momentum_x: 2.0559e+00 0.1% 2.1431e+09 0.0% 1.840e+02 0.0% 1.151e+01 0.0% 2.800e+01 0.2% 3: momentum_y: 1.9997e+00 0.1% 2.1583e+09 0.0% 1.840e+02 0.0% 1.167e+01 0.0% 2.800e+01 0.2% 4: momentum_z: 2.7697e+03 90.6% 6.7873e+12 97.2% 2.312e+06 97.2% 8.280e+04 97.1% 1.258e+04 90.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 1053 1.0 3.4221e+01 1.3 5.37e+09 1.3 4.2e+04 1.0e+05 0.0e+00 1 1 2 2 0 44 53 65 75 0 2962 MatMultAdd 156 1.0 1.6699e+00 1.5 3.58e+08 1.5 6.1e+03 1.7e+04 0.0e+00 0 0 0 0 0 2 3 9 2 0 3677 MatMultTranspose 156 1.0 4.1421e+00 3.5 3.58e+08 1.5 6.1e+03 1.7e+04 0.0e+00 0 0 0 0 0 3 3 9 2 0 1482 MatSolve 78 0.0 4.3559e-04 0.0 7.45e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 171 MatLUFactorSym 1 1.0 4.1962e-05 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 3.6955e-05 4.1 6.48e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 175 MatConvert 4 1.0 1.7373e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.8639e-01 1.7 3.05e+07 1.4 1.7e+02 9.8e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 2980 MatAssemblyBegin 69 1.0 4.5507e+00 3.2 0.00e+00 0.0 5.3e+02 1.6e+04 7.4e+01 0 0 0 0 1 4 0 1 0 6 0 MatAssemblyEnd 69 1.0 1.4305e+00 1.2 0.00e+00 0.0 2.6e+03 1.3e+04 2.0e+02 0 0 0 0 1 2 0 4 1 15 0 MatGetRow 1821330 1.1 6.5450e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 0.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 7.3195e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.3e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 5.3315e-01 1.3 0.00e+00 0.0 3.1e+03 4.7e+04 1.8e+02 0 0 0 0 1 1 0 5 2 14 0 MatAXPY 4 1.0 3.9307e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 1.1955e+00 1.0 2.13e+07 1.3 1.1e+03 4.4e+04 9.6e+01 0 0 0 0 1 2 0 2 1 7 333 MatMatMultSym 4 1.0 9.2042e-01 1.0 0.00e+00 0.0 9.7e+02 3.5e+04 8.8e+01 0 0 0 0 1 1 0 1 1 7 0 MatMatMultNum 4 1.0 2.8186e-01 1.0 2.13e+07 1.3 1.7e+02 9.8e+04 8.0e+00 0 0 0 0 0 0 0 0 0 1 1412 MatPtAP 4 1.0 4.3431e+00 1.0 8.45e+08 2.7 2.2e+03 9.0e+04 1.1e+02 0 0 0 0 1 6 5 3 3 8 1978 MatPtAPSymbolic 4 1.0 2.4485e+00 1.0 0.00e+00 0.0 2.0e+03 7.5e+04 1.0e+02 0 0 0 0 1 4 0 3 2 8 0 MatPtAPNumeric 4 1.0 1.8946e+00 1.0 8.45e+08 2.7 2.4e+02 2.1e+05 8.0e+00 0 0 0 0 0 3 5 0 1 1 4534 MatTrnMatMult 4 1.0 9.8843e+00 1.0 1.03e+09 3.2 1.1e+03 5.9e+05 1.2e+02 0 0 0 0 1 15 12 2 11 9 2291 MatGetLocalMat 20 1.0 5.3209e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 12 1.0 2.1986e-01 2.3 0.00e+00 0.0 1.2e+03 1.4e+05 1.6e+01 0 0 0 0 0 0 0 2 3 1 0 MatGetSymTrans 8 1.0 5.6299e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 79 1.0 9.2357e-01 2.1 2.00e+08 1.1 0.0e+00 0.0e+00 7.9e+01 0 0 0 0 1 1 2 0 0 6 4984 KSPSetUp 11 1.0 1.5080e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 KSPSolve 1 1.0 6.5990e+01 1.0 9.22e+09 1.2 6.5e+04 9.0e+04 1.3e+03 2 3 3 3 9 97100100100 99 2889 VecDot 38 1.0 3.3833e+00 9.7 6.22e+07 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 0 2 1 0 0 3 428 VecDotNorm2 19 1.0 3.3915e+00 7.4 1.24e+08 1.0 0.0e+00 0.0e+00 5.7e+01 0 0 0 0 0 2 2 0 0 4 854 VecMDot 79 1.0 7.9980e-01 3.8 1.00e+08 1.1 0.0e+00 0.0e+00 7.9e+01 0 0 0 0 1 1 1 0 0 6 2878 VecNorm 142 1.0 2.0564e+0011.8 5.28e+07 1.0 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 1 1 1 0 0 11 595 VecScale 746 1.0 9.4111e-01 3.6 1.52e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 3713 VecCopy 201 1.0 2.6442e-01 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 741 1.0 3.9763e-01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1291 1.0 3.2086e+00 4.0 5.70e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 4082 VecAYPX 1248 1.0 3.0970e+00 3.1 3.55e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 4 0 0 0 2635 VecAXPBYCZ 38 1.0 1.1318e+00 3.5 1.24e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 2560 VecWAXPY 38 1.0 1.0047e+00 3.5 6.22e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 1442 VecMAXPY 122 1.0 3.1358e-01 2.1 1.18e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 8674 VecAssemblyBegin 72 1.0 2.3761e-0110.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+02 0 0 0 0 2 0 0 0 0 16 0 VecAssemblyEnd 72 1.0 1.7858e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 980 1.0 3.3524e+00 3.6 2.23e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 3 0 0 0 1529 VecScatterBegin 1443 1.0 3.2857e-01 5.7 0.00e+00 0.0 5.9e+04 8.4e+04 0.0e+00 0 0 2 2 0 0 0 91 84 0 0 VecScatterEnd 1443 1.0 2.1979e+0125.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0 VecSetRandom 4 1.0 4.0149e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 122 1.0 1.3411e+0013.5 3.01e+07 1.1 0.0e+00 0.0e+00 1.2e+02 0 0 0 0 1 0 0 0 0 9 515 PCSetUp 2 1.0 2.2811e+01 1.0 1.83e+09 1.2 1.2e+04 1.2e+05 1.1e+03 1 1 1 1 8 34 22 19 24 80 1839 PCSetUpOnBlocks 39 1.0 2.9802e-04 1.9 6.48e+03 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 22 PCApply 39 1.0 3.5861e+01 1.1 6.76e+09 1.3 5.1e+04 7.9e+04 1.3e+02 1 2 2 2 1 51 68 78 68 10 3622 PCGAMGgraph_AGG 4 1.0 2.9758e+00 1.0 2.13e+07 1.3 4.9e+02 4.9e+04 7.6e+01 0 0 0 0 1 4 0 1 0 6 134 PCGAMGcoarse_AGG 4 1.0 1.0956e+01 1.0 1.03e+09 3.2 5.2e+03 1.8e+05 3.8e+02 0 0 0 0 3 16 12 8 16 29 2067 PCGAMGProl_AGG 4 1.0 1.1267e+00 1.0 0.00e+00 0.0 1.3e+03 5.5e+04 1.1e+02 0 0 0 0 1 2 0 2 1 9 0 PCGAMGPOpt_AGG 4 1.0 3.4058e+00 1.0 4.93e+08 1.2 2.8e+03 7.6e+04 2.1e+02 0 0 0 0 2 5 5 4 4 16 3027 --- Event Stage 2: momentum_x MatMult 2 1.0 9.7961e-02 1.8 2.09e+07 1.1 9.2e+01 2.4e+05 0.0e+00 0 0 0 0 0 4 23 50 80 0 4962 MatSolve 3 1.0 1.1522e-01 1.7 3.10e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 34 0 0 0 6257 MatLUFactorNum 1 1.0 1.6440e-01 1.3 1.73e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 7 19 0 0 0 2448 MatILUFactorSym 1 1.0 1.3943e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 5 0 0 0 4 0 MatAssemblyBegin 1 1.0 4.3930e-014067.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 11 0 0 0 7 0 MatAssemblyEnd 1 1.0 2.0642e-01 1.2 0.00e+00 0.0 9.2e+01 5.9e+04 8.0e+00 0 0 0 0 0 10 0 50 20 29 0 MatGetRowIJ 1 1.0 5.0068e-06 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.8502e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 5.3131e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 6.5504e-01 1.0 9.21e+07 1.1 9.2e+01 2.4e+05 1.2e+01 0 0 0 0 0 32100 50 80 43 3272 VecDot 2 1.0 5.4927e-0210.2 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 1388 VecDotNorm2 1 1.0 5.2273e-02 6.6 6.54e+06 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 1 7 0 0 11 2917 VecNorm 2 1.0 1.5215e-0157.3 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 4 4 0 0 7 501 VecCopy 2 1.0 8.7559e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.7282e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 2.3268e-02 3.2 6.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 6553 VecWAXPY 2 1.0 1.9122e-02 2.8 3.27e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 3987 VecAssemblyBegin 2 1.0 4.0814e-02200.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 1 0 0 0 21 0 VecAssemblyEnd 2 1.0 1.4067e-05 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 1.3988e-03 3.6 0.00e+00 0.0 9.2e+01 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 4.9311e-0226.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 2 1.0 3.1949e-01 1.6 1.73e+07 1.1 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 13 19 0 0 18 1260 PCSetUpOnBlocks 1 1.0 3.1934e-01 1.6 1.73e+07 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 13 19 0 0 11 1260 PCApply 3 1.0 1.2085e-01 1.7 3.10e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 5966 --- Event Stage 3: momentum_y MatMult 2 1.0 9.5242e-02 1.7 2.11e+07 1.1 9.2e+01 2.4e+05 0.0e+00 0 0 0 0 0 4 23 50 80 0 5153 MatSolve 3 1.0 1.1227e-01 1.7 3.13e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 6483 MatLUFactorNum 1 1.0 1.7762e-01 1.4 1.75e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 8 19 0 0 0 2286 MatILUFactorSym 1 1.0 1.3570e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 5 0 0 0 4 0 MatAssemblyBegin 1 1.0 3.2052e-013021.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 8 0 0 0 7 0 MatAssemblyEnd 1 1.0 1.9790e-01 1.2 0.00e+00 0.0 9.2e+01 6.0e+04 8.0e+00 0 0 0 0 0 9 0 50 20 29 0 MatGetRowIJ 1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.7504e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 5.5541e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 6.6664e-01 1.0 9.28e+07 1.1 9.2e+01 2.4e+05 1.2e+01 0 0 0 0 0 33100 50 80 43 3238 VecDot 2 1.0 5.4068e-02 9.8 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 4 0 0 7 1410 VecDotNorm2 1 1.0 5.0182e-02 6.4 6.54e+06 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 1 7 0 0 11 3038 VecNorm 2 1.0 1.5866e-0121.8 3.27e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 4 4 0 0 7 480 VecCopy 2 1.0 9.0349e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.6930e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 2.2837e-02 3.0 6.54e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 6676 VecWAXPY 2 1.0 1.9680e-02 2.7 3.27e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 3874 VecAssemblyBegin 2 1.0 4.1746e-02243.9 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 1 0 0 0 21 0 VecAssemblyEnd 2 1.0 1.3113e-05 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 1.3828e-03 3.6 0.00e+00 0.0 9.2e+01 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 5.0416e-0228.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 2 1.0 3.2991e-01 1.6 1.75e+07 1.1 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 14 19 0 0 18 1231 PCSetUpOnBlocks 1 1.0 3.2973e-01 1.6 1.75e+07 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 14 19 0 0 11 1231 PCApply 3 1.0 1.1918e-01 1.7 3.13e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 34 0 0 0 6107 --- Event Stage 4: momentum_z MatMult 44415 1.0 1.9922e+03 1.5 2.28e+11 1.3 1.8e+06 1.1e+05 0.0e+00 52 62 75 93 0 58 64 77 95 0 2169 MatMultAdd 6796 1.0 1.4178e+02 2.6 1.56e+10 1.5 2.7e+05 1.7e+04 0.0e+00 3 4 11 2 0 3 4 11 2 0 1887 MatMultTranspose 6796 1.0 3.1148e+02 5.0 1.56e+10 1.5 2.7e+05 1.7e+04 0.0e+00 6 4 11 2 0 6 4 11 2 0 859 MatSolve 3833 8.8 2.8625e+01 2.7 4.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 3676 MatLUFactorNum 145 1.0 3.1267e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1878 MatILUFactorSym 1 1.0 1.3837e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 145 1.0 2.9783e+0114.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 1 0 0 0 2 1 0 0 0 2 0 MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.9623e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 1699 1.0 6.9188e-02 1.3 1.55e+05 0.0 0.0e+00 0.0e+00 1.7e+03 0 0 0 0 12 0 0 0 0 14 2 KSPSetUp 290 1.0 6.0805e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 194 1.0 2.6100e+03 1.0 3.41e+11 1.2 2.3e+06 8.5e+04 1.1e+04 85 97 97 97 80 94100100100 88 2600 VecDot 1940 1.0 2.6695e+0212.3 3.17e+09 1.0 0.0e+00 0.0e+00 1.9e+03 5 1 0 0 14 6 1 0 0 15 277 VecDotNorm2 970 1.0 2.6302e+0210.2 6.35e+09 1.0 0.0e+00 0.0e+00 2.9e+03 5 2 0 0 21 6 2 0 0 23 562 VecMDot 1699 1.0 5.7786e-02 1.7 7.65e+04 0.0 0.0e+00 0.0e+00 1.7e+03 0 0 0 0 12 0 0 0 0 14 1 VecNorm 4562 1.0 1.2351e+0213.8 1.90e+09 1.0 0.0e+00 0.0e+00 4.6e+03 2 1 0 0 33 2 1 0 0 36 359 VecScale 30582 1.0 3.7316e+01 2.8 6.19e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 3810 VecCopy 8883 1.0 1.9754e+01 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 29901 1.0 2.2141e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 56067 1.0 1.5246e+02 3.7 2.48e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 8 0 0 0 3 8 0 0 0 3731 VecAYPX 54368 1.0 1.8245e+02 3.3 1.55e+10 1.1 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 1948 VecAXPBYCZ 1940 1.0 4.4522e+01 2.5 6.35e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 3322 VecWAXPY 1940 1.0 4.2226e+01 2.2 3.17e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 1751 VecMAXPY 3398 1.0 7.1802e-03 2.8 1.56e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 22 VecAssemblyBegin 388 1.0 1.0391e+01 7.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03 0 0 0 0 8 0 0 0 0 9 0 VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 40776 1.0 1.8958e+02 3.9 9.28e+09 1.1 0.0e+00 0.0e+00 0.0e+00 3 3 0 0 0 3 3 0 0 0 1125 VecScatterBegin 58007 1.0 1.7609e+01 5.9 0.00e+00 0.0 2.3e+06 8.5e+04 0.0e+00 0 0 97 97 0 0 0100100 0 0 VecScatterEnd 58007 1.0 1.3547e+0315.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 29 0 0 0 0 32 0 0 0 0 0 VecNormalize 3398 1.0 6.5100e+01772.4 2.34e+05 0.0 0.0e+00 0.0e+00 3.4e+03 0 0 0 0 24 0 0 0 0 27 0 PCSetUp 290 1.0 3.1397e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 1 1 0 0 0 1871 PCSetUpOnBlocks 1844 1.0 3.1398e+01 1.7 2.53e+09 1.1 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 1 1 0 0 0 1871 PCApply 2134 1.0 2.1924e+03 1.2 2.99e+11 1.3 2.2e+06 7.9e+04 5.1e+03 65 83 93 86 36 72 85 96 89 41 2629 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 12 56 1076276336 0 Krylov Solver 4 13 31728 0 Vector 8 85 248594232 0 Vector Scatter 0 12 12720 0 Index Set 0 16 9829416 0 Preconditioner 0 13 12540 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 117 76 734590640 0 Matrix Coarsen 4 4 2480 0 Krylov Solver 10 4 120512 0 Vector 225 175 200940576 0 Vector Scatter 31 22 23320 0 Index Set 81 74 765564 0 Preconditioner 11 4 3456 0 PetscRandom 4 4 2464 0 --- Event Stage 2: momentum_x Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 120488 0 Preconditioner 2 0 0 0 --- Event Stage 3: momentum_y Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122104 0 Preconditioner 2 0 0 0 --- Event Stage 4: momentum_z Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122096 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 Average time for MPI_Barrier(): 1.45912e-05 Average time for zero size MPI_Send(): 1.23382e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Mon Oct 1 11:36:09 2012 on hpc12 Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.3-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-02 with 48 processors, by wtay Wed Oct 3 21:04:02 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 1.721e+03 1.00353 1.718e+03 Objects: 5.680e+02 1.00000 5.680e+02 Flops: 1.739e+11 1.26420 1.417e+11 6.803e+12 Flops/sec: 1.010e+08 1.26365 8.249e+07 3.960e+09 MPI Messages: 1.686e+05 3.29646 1.101e+05 5.285e+06 MPI Message Lengths: 8.665e+09 2.00778 7.622e+04 4.029e+11 MPI Reductions: 1.377e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.1566e+02 12.6% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 3.600e+01 0.3% 1: poisson: 5.6307e+01 3.3% 2.1388e+11 3.1% 1.812e+05 3.4% 2.545e+03 3.3% 1.443e+03 10.5% 2: momentum_x: 1.8503e+00 0.1% 2.1267e+09 0.0% 3.760e+02 0.0% 1.058e+01 0.0% 2.800e+01 0.2% 3: momentum_y: 1.5023e+00 0.1% 2.1417e+09 0.0% 3.760e+02 0.0% 1.073e+01 0.0% 2.800e+01 0.2% 4: momentum_z: 1.4426e+03 84.0% 6.5844e+12 96.8% 5.103e+06 96.6% 7.366e+04 96.6% 1.223e+04 88.8% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 1209 1.0 2.7780e+01 1.3 3.16e+09 1.3 1.2e+05 8.9e+04 0.0e+00 1 2 2 3 0 43 55 64 77 0 4208 MatMultAdd 180 1.0 1.5880e+00 1.8 2.18e+08 1.5 1.4e+04 1.8e+04 0.0e+00 0 0 0 0 0 2 3 7 2 0 4488 MatMultTranspose 180 1.0 3.2210e+00 3.9 2.18e+08 1.5 1.4e+04 1.8e+04 0.0e+00 0 0 0 0 0 3 3 7 2 0 2212 MatSolve 90 0.0 6.0534e-04 0.0 1.18e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 196 MatLUFactorSym 1 1.0 6.1035e-05 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 5.4121e-05 6.1 1.04e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 191 MatConvert 4 1.0 1.3334e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.3946e-01 1.7 1.58e+07 1.4 4.0e+02 8.3e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 4001 MatAssemblyBegin 69 1.0 4.9522e+00 2.5 0.00e+00 0.0 1.2e+03 1.4e+04 7.4e+01 0 0 0 0 1 6 0 1 0 5 0 MatAssemblyEnd 69 1.0 1.4642e+00 1.1 0.00e+00 0.0 6.7e+03 1.0e+04 2.0e+02 0 0 0 0 1 2 0 4 1 14 0 MatGetRow 961340 1.1 4.5354e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 0.0 1.4782e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 7.7963e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 5.2552e-01 1.2 0.00e+00 0.0 1.9e+04 1.8e+04 2.8e+02 0 0 0 0 2 1 0 10 2 20 0 MatAXPY 4 1.0 4.2417e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 1.0365e+00 1.0 1.10e+07 1.4 2.8e+03 3.7e+04 9.6e+01 0 0 0 0 1 2 0 2 1 7 386 MatMatMultSym 4 1.0 7.9752e-01 1.1 0.00e+00 0.0 2.4e+03 2.9e+04 8.8e+01 0 0 0 0 1 1 0 1 1 6 0 MatMatMultNum 4 1.0 2.5070e-01 1.1 1.10e+07 1.4 4.0e+02 8.3e+04 8.0e+00 0 0 0 0 0 0 0 0 0 1 1594 MatPtAP 4 1.0 3.2046e+00 1.0 4.77e+08 3.1 5.9e+03 6.9e+04 1.1e+02 0 0 0 0 1 6 4 3 3 7 2670 MatPtAPSymbolic 4 1.0 1.9027e+00 1.1 0.00e+00 0.0 5.3e+03 5.7e+04 1.0e+02 0 0 0 0 1 3 0 3 2 7 0 MatPtAPNumeric 4 1.0 1.3268e+00 1.0 4.77e+08 3.1 6.0e+02 1.7e+05 8.0e+00 0 0 0 0 0 2 4 0 1 1 6450 MatTrnMatMult 4 1.0 8.2181e+00 1.0 5.30e+08 4.6 2.7e+03 4.7e+05 1.2e+02 0 0 0 0 1 15 10 1 10 8 2722 MatGetLocalMat 20 1.0 3.5593e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 12 1.0 2.3268e-01 2.4 0.00e+00 0.0 2.8e+03 1.2e+05 1.6e+01 0 0 0 0 0 0 0 2 3 1 0 MatGetSymTrans 8 1.0 4.9805e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 85 1.0 6.8444e-01 1.7 1.06e+08 1.1 0.0e+00 0.0e+00 8.5e+01 0 0 0 0 1 1 2 0 0 6 6731 KSPSetUp 11 1.0 1.4524e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 KSPSolve 1 1.0 5.5045e+01 1.0 5.31e+09 1.3 1.8e+05 7.4e+04 1.4e+03 3 3 3 3 10 98100100100 99 3886 VecDot 44 1.0 2.4764e+00 3.6 3.73e+07 1.1 0.0e+00 0.0e+00 4.4e+01 0 0 0 0 0 3 1 0 0 3 677 VecDotNorm2 22 1.0 2.5759e+00 4.2 7.47e+07 1.1 0.0e+00 0.0e+00 6.6e+01 0 0 0 0 0 3 2 0 0 5 1302 VecMDot 85 1.0 5.4579e-01 2.1 5.29e+07 1.1 0.0e+00 0.0e+00 8.5e+01 0 0 0 0 1 1 1 0 0 6 4221 VecNorm 157 1.0 2.4986e+0011.5 3.01e+07 1.1 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 1 0 0 11 535 VecScale 854 1.0 5.2014e-01 2.8 9.18e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 7690 VecCopy 231 1.0 2.2006e-01 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 843 1.0 2.7842e-01 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 1489 1.0 1.6388e+00 2.8 3.47e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 9226 VecAYPX 1440 1.0 2.2286e+00 2.6 2.16e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 4 0 0 0 4229 VecAXPBYCZ 44 1.0 6.8125e-01 2.2 7.47e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 2 0 0 0 4924 VecWAXPY 44 1.0 7.3601e-01 2.3 3.73e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 1 0 0 0 2279 VecMAXPY 134 1.0 2.0552e-01 1.9 6.25e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 13246 VecAssemblyBegin 97 1.0 3.6385e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.8e+02 0 0 0 0 2 1 0 0 0 20 0 VecAssemblyEnd 97 1.0 3.5381e-04 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 1124 1.0 2.1477e+00 2.4 1.35e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 3 0 0 0 2740 VecScatterBegin 1672 1.0 4.9030e-01 3.5 0.00e+00 0.0 1.7e+05 7.0e+04 0.0e+00 0 0 3 3 0 1 0 92 86 0 0 VecScatterEnd 1672 1.0 1.9463e+0113.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 16 0 0 0 0 0 VecSetRandom 4 1.0 4.0139e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 134 1.0 2.1754e+0016.8 1.59e+07 1.1 0.0e+00 0.0e+00 1.3e+02 0 0 0 0 1 2 0 0 0 9 318 PCSetUp 2 1.0 1.8591e+01 1.0 9.41e+08 1.3 4.2e+04 7.0e+04 1.2e+03 1 1 1 1 8 33 19 23 22 80 2241 PCSetUpOnBlocks 45 1.0 6.5732e-04 3.5 1.04e+04 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 16 PCApply 45 1.0 2.9939e+01 1.1 4.01e+09 1.3 1.4e+05 7.0e+04 1.5e+02 2 2 3 2 1 51 70 75 71 10 5025 PCGAMGgraph_AGG 4 1.0 2.3984e+00 1.0 1.10e+07 1.4 1.2e+03 4.2e+04 7.6e+01 0 0 0 0 1 4 0 1 0 5 167 PCGAMGcoarse_AGG 4 1.0 9.2185e+00 1.0 5.30e+08 4.6 2.4e+04 7.9e+04 4.8e+02 1 0 0 0 3 16 10 13 14 33 2427 PCGAMGProl_AGG 4 1.0 9.8128e-01 1.0 0.00e+00 0.0 3.2e+03 4.6e+04 1.1e+02 0 0 0 0 1 2 0 2 1 8 0 PCGAMGPOpt_AGG 4 1.0 2.8071e+00 1.0 2.53e+08 1.2 6.8e+03 6.4e+04 2.1e+02 0 0 0 0 2 5 5 4 3 15 3682 --- Event Stage 2: momentum_x MatMult 2 1.0 1.0542e-01 2.0 1.08e+07 1.2 1.9e+02 2.4e+05 0.0e+00 0 0 0 0 0 3 23 50 80 0 4611 MatSolve 3 1.0 8.1705e-02 2.0 1.59e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 33 0 0 0 8719 MatLUFactorNum 1 1.0 1.4021e-01 2.1 8.82e+06 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 19 0 0 0 2814 MatILUFactorSym 1 1.0 1.3817e-01 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 4 0 0 0 4 0 MatAssemblyBegin 1 1.0 7.5712e-014946.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 27 0 0 0 7 0 MatAssemblyEnd 1 1.0 1.8089e-01 1.2 0.00e+00 0.0 1.9e+02 5.9e+04 8.0e+00 0 0 0 0 0 9 0 50 20 29 0 MatGetRowIJ 1 1.0 1.0014e-0510.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.1192e-02 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 7 0 KSPSetUp 2 1.0 6.2056e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 6.0202e-01 1.0 4.74e+07 1.1 1.9e+02 2.4e+05 1.2e+01 0 0 0 0 0 32100 50 80 43 3533 VecDot 2 1.0 4.6937e-02 3.8 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 2 4 0 0 7 1624 VecDotNorm2 1 1.0 4.7157e-02 6.1 3.39e+06 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 2 7 0 0 11 3233 VecNorm 2 1.0 2.2409e-0134.8 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 9 4 0 0 7 340 VecCopy 2 1.0 9.5789e-03 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.3558e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPBYCZ 2 1.0 2.1303e-02 3.6 3.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 7157 VecWAXPY 2 1.0 1.6192e-02 2.7 1.70e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 4708 VecAssemblyBegin 2 1.0 5.5579e-02156.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 2 0 0 0 21 0 VecAssemblyEnd 2 1.0 1.5020e-05 7.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 1.5790e-03 2.4 0.00e+00 0.0 1.9e+02 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 5.5989e-0228.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 2 1.0 3.0002e-01 2.4 8.82e+06 1.2 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 9 19 0 0 18 1315 PCSetUpOnBlocks 1 1.0 2.9975e-01 2.4 8.82e+06 1.2 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 9 19 0 0 11 1316 PCApply 3 1.0 8.5135e-02 1.9 1.59e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 33 0 0 0 8367 --- Event Stage 3: momentum_y MatMult 2 1.0 1.0478e-01 2.0 1.09e+07 1.2 1.9e+02 2.4e+05 0.0e+00 0 0 0 0 0 4 23 50 80 0 4684 MatSolve 3 1.0 7.1359e-02 1.6 1.61e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 34 0 0 0 10078 MatLUFactorNum 1 1.0 1.3938e-01 1.9 8.90e+06 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 6 19 0 0 0 2856 MatILUFactorSym 1 1.0 1.1233e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 5 0 0 0 4 0 MatAssemblyBegin 1 1.0 3.2594e-0124.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 13 0 0 0 7 0 MatAssemblyEnd 1 1.0 1.8197e-01 1.1 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00 0 0 0 0 0 11 0 50 20 29 0 MatGetRowIJ 1 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.0951e-02 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 1 0 0 0 7 0 KSPSetUp 2 1.0 6.4847e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 KSPSolve 1 1.0 5.6900e-01 1.0 4.78e+07 1.1 1.9e+02 2.4e+05 1.2e+01 0 0 0 0 0 37100 50 80 43 3764 VecDot 2 1.0 4.5465e-02 3.5 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 2 4 0 0 7 1677 VecDotNorm2 1 1.0 5.0043e-02 3.8 3.39e+06 1.1 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 3 7 0 0 11 3047 VecNorm 2 1.0 1.7585e-0133.5 1.70e+06 1.1 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 8 4 0 0 7 434 VecCopy 2 1.0 6.0210e-03 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 7 1.0 1.7671e-02 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 2 1.0 1.4564e-02 2.4 3.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 10469 VecWAXPY 2 1.0 1.6496e-02 2.7 1.70e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 4621 VecAssemblyBegin 2 1.0 4.9739e-02159.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 2 0 0 0 21 0 VecAssemblyEnd 2 1.0 1.7881e-05 6.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 2 1.0 1.1569e-0219.1 0.00e+00 0.0 1.9e+02 2.4e+05 0.0e+00 0 0 0 0 0 0 0 50 80 0 0 VecScatterEnd 2 1.0 5.5051e-0223.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 2 1.0 2.7300e-01 2.1 8.90e+06 1.2 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 12 19 0 0 18 1458 PCSetUpOnBlocks 1 1.0 2.7282e-01 2.1 8.90e+06 1.2 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 12 19 0 0 11 1459 PCApply 3 1.0 7.5869e-02 1.6 1.61e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 34 0 0 0 9479 --- Event Stage 4: momentum_z MatMult 42907 1.0 9.8638e+02 1.3 1.13e+11 1.3 4.1e+06 9.0e+04 0.0e+00 51 62 78 92 0 61 64 81 95 0 4250 MatMultAdd 6564 1.0 7.0460e+01 2.2 7.96e+09 1.5 4.9e+05 1.8e+04 0.0e+00 2 4 9 2 0 3 4 10 2 0 3688 MatMultTranspose 6564 1.0 1.2860e+02 4.9 7.96e+09 1.5 4.9e+05 1.8e+04 0.0e+00 3 4 9 2 0 4 4 10 2 0 2021 MatSolve 3717 8.5 1.1711e+01 1.4 2.33e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 8879 MatLUFactorNum 145 1.0 1.4473e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 3979 MatILUFactorSym 1 1.0 1.2902e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 145 1.0 2.5208e+01 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 1 0 0 0 2 1 0 0 0 2 0 MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.9673e-02 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 1641 1.0 1.7774e-01 1.4 1.76e+05 0.0 0.0e+00 0.0e+00 1.6e+03 0 0 0 0 12 0 0 0 0 13 1 KSPSetUp 290 1.0 7.1497e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 194 1.0 1.3382e+03 1.0 1.68e+11 1.3 5.1e+06 7.6e+04 1.1e+04 78 97 97 97 78 93100100100 88 4920 VecDot 1882 1.0 8.1310e+01 5.0 1.60e+09 1.1 0.0e+00 0.0e+00 1.9e+03 3 1 0 0 14 3 1 0 0 15 882 VecDotNorm2 941 1.0 8.2061e+01 4.1 3.19e+09 1.1 0.0e+00 0.0e+00 2.8e+03 3 2 0 0 21 3 2 0 0 23 1748 VecMDot 1641 1.0 1.6827e-01 1.5 8.70e+04 0.0 0.0e+00 0.0e+00 1.6e+03 0 0 0 0 12 0 0 0 0 13 1 VecNorm 4417 1.0 6.4342e+01 9.1 9.63e+08 1.1 0.0e+00 0.0e+00 4.4e+03 1 1 0 0 32 1 1 0 0 36 672 VecScale 29538 1.0 1.9634e+01 3.2 3.16e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 7001 VecCopy 8593 1.0 1.0218e+01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 28915 1.0 1.3080e+01 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 54153 1.0 6.1113e+01 3.0 1.26e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 8 0 0 0 3 8 0 0 0 8997 VecAYPX 52512 1.0 8.6173e+01 2.7 7.89e+09 1.1 0.0e+00 0.0e+00 0.0e+00 4 5 0 0 0 4 5 0 0 0 3988 VecAXPBYCZ 1882 1.0 2.8966e+01 2.1 3.19e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 4953 VecWAXPY 1882 1.0 2.8427e+01 2.1 1.60e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2524 VecMAXPY 3282 1.0 6.8157e-03 3.1 1.77e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 26 VecAssemblyBegin 388 1.0 8.7212e+0011.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+03 0 0 0 0 8 0 0 0 0 10 0 VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 39384 1.0 8.7410e+01 2.7 4.73e+09 1.1 0.0e+00 0.0e+00 0.0e+00 3 3 0 0 0 4 3 0 0 0 2359 VecScatterBegin 56035 1.0 2.0136e+01 4.1 0.00e+00 0.0 5.1e+06 7.6e+04 0.0e+00 1 0 97 97 0 1 0100100 0 0 VecScatterEnd 56035 1.0 6.7774e+0213.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 22 0 0 0 0 0 VecNormalize 3282 1.0 4.7531e+01301.1 2.66e+05 0.0 0.0e+00 0.0e+00 3.3e+03 0 0 0 0 24 0 0 0 0 27 0 PCSetUp 290 1.0 1.4621e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 1 1 0 0 0 3939 PCSetUpOnBlocks 1786 1.0 1.4621e+01 1.3 1.29e+09 1.2 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 1 1 0 0 0 3939 PCApply 2076 1.0 1.1022e+03 1.1 1.48e+11 1.3 4.9e+06 7.0e+04 4.9e+03 61 82 93 86 36 73 85 97 89 40 5071 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 12 56 568064932 0 Krylov Solver 4 13 31728 0 Vector 8 85 129979032 0 Vector Scatter 0 12 12720 0 Index Set 0 16 5102648 0 Preconditioner 0 13 12540 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 117 76 368468980 0 Matrix Coarsen 4 4 2480 0 Krylov Solver 10 4 120512 0 Vector 225 175 107847696 0 Vector Scatter 31 22 23320 0 Index Set 81 74 737472 0 Preconditioner 11 4 3456 0 PetscRandom 4 4 2464 0 --- Event Stage 2: momentum_x Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 120488 0 Preconditioner 2 0 0 0 --- Event Stage 3: momentum_y Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122104 0 Preconditioner 2 0 0 0 --- Event Stage 4: momentum_z Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122096 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.19209e-07 Average time for MPI_Barrier(): 4.45843e-05 Average time for zero size MPI_Send(): 1.93516e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Mon Oct 1 11:36:09 2012 on hpc12 Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.3-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- From jedbrown at mcs.anl.gov Wed Oct 3 16:58:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Oct 2012 16:58:22 -0500 Subject: [petsc-users] PetscFinalize() In-Reply-To: References: Message-ID: On Wed, Oct 3, 2012 at 11:58 AM, Eleni Siampli wrote: > It doen't work good. I have created a dll for solving the eigenPorblem > (eigenSolver.dll) and I included it in my current project. > lets say that eigenSolver is the name of my class.. What I use is: > eigensolver eig; > for a lot of iterations and works fine. I call only one the > PetscInitialize(); > > I want also to use: > eigensolver eig1; > eigensolver eig2; > > I call only once PetscInitialize(); for each eig1 and eig 2. > It is much better to call PetscInitialize only once, ever. There are lots of ways to set that up in your code. I would use a static variable and register a finalize routine with my main. -------------- next part -------------- An HTML attachment was scrubbed... URL: From siaeleni at hotmail.com Wed Oct 3 17:12:34 2012 From: siaeleni at hotmail.com (Eleni Siampli) Date: Thu, 4 Oct 2012 01:12:34 +0300 Subject: [petsc-users] PetscFinalize() In-Reply-To: <472BDE33-C313-4BEF-83B1-ED75CB2D273B@mcs.anl.gov> References: , , , , , , , , <472BDE33-C313-4BEF-83B1-ED75CB2D273B@mcs.anl.gov> Message-ID: Thanks a lot. The problem solved. Helen > From: bsmith at mcs.anl.gov > Date: Wed, 3 Oct 2012 14:16:07 -0500 > To: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] PetscFinalize() > > > This error occurs if more than 250 functions are registered with PetscRegisterFinalize(). Under normal usage that should never happen. > > Likely something is calling PetscRegisterFinalize() unprotected, that is, calling it many times with the same function to register. Likely this is in SLEPc. > > The other possibility is some strange interaction with .dll on Microsoft Windows? > > First I would start your program in the debugger, put a break point in PetscRegisterFinalize() and then run the program. Each time it breaks in PetscRegisterFinalize() note where it is called from. Under normal circumstances PETSc should call > PetscRegisterFinalize() around 10 times. If you see it called 30 times note what functions are calling it over and over again and send back that information. > > Barry > > > > On Oct 3, 2012, at 11:58 AM, Eleni Siampli wrote: > > > > > Hello again, > > > > It doen't work good. I have created a dll for solving the eigenPorblem (eigenSolver.dll) and I included it in my current project. > > lets say that eigenSolver is the name of my class.. What I use is: > > eigensolver eig; > > for a lot of iterations and works fine. I call only one the PetscInitialize(); > > > > I want also to use: > > eigensolver eig1; > > eigensolver eig2; > > > > I call only once PetscInitialize(); for each eig1 and eig 2. > > > > I have the same error as before. > > > > Why does it give me the same error?I call only once the PetscInitialize(). > > Thanks a lot in advance, > > Helen > > > > Date: Fri, 28 Sep 2012 20:23:20 -0400 > > From: knepley at gmail.com > > To: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] PetscFinalize() > > > > On Fri, Sep 28, 2012 at 5:09 PM, Eleni Siampli wrote: > > > > Thank you for the answer. Problem solved, but now I have another one: > > > > After a lot of iterations (around 246) It gives me the following error: > > > > > > [0]PETSC ERROR: Petsc has generated inconsistent data! > > [0]PETSC ERROR: No more room in array, limit 256 > > recompile src/sys/objects/destroy.c with larger value for MAXREGFIN > > > > Do you have any idea about this problem? > > > > You are also calling PetscInitialize() a bunch of times. Its intended for just one. > > > > Matt > > > > > > Thanks, > > Helen > > Date: Fri, 28 Sep 2012 11:17:51 -0400 > > From: knepley at gmail.com > > To: petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] PetscFinalize() > > > > On Fri, Sep 28, 2012 at 11:16 AM, Eleni Siampli wrote: > > Hey all, > > > > I would like to solve the eigenvalue problem. > > My code is working for 2 loops and give the right values, but after two loops it gives me the following error: > > > > [0]PETSC ERROR: PetscFinalize() line 1221 in src/sys/objects/C:\cygwin\home\liyi > > 0000\PETSC-~1.2-P\src\sys\objects\pinit.c > > [0]PETSC ERROR: SlepcFinalize() line 224 in src/sys/C:\cygwin\home\liyi0000\SLEP > > C-~1.2-P\src\sys\slepcinit.c > > Options have not been enabled. > > You might have forgotten to call PetscInitialize(). > > > > Do you have any idea what this error mean? > > > > You are only intended to call PetscFinalize() once, at the end of your program. > > > > Matt > > > > > > Thank you in advance, > > Helen > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Oct 3 18:53:48 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 3 Oct 2012 18:53:48 -0500 Subject: [petsc-users] PetscFinalize() In-Reply-To: References: Message-ID: <205C27BA-1161-4EAC-81D3-EA9B889E577E@mcs.anl.gov> On Oct 3, 2012, at 4:58 PM, Jed Brown wrote: > On Wed, Oct 3, 2012 at 11:58 AM, Eleni Siampli wrote: > It doen't work good. I have created a dll for solving the eigenPorblem (eigenSolver.dll) and I included it in my current project. > lets say that eigenSolver is the name of my class.. What I use is: > eigensolver eig; > for a lot of iterations and works fine. I call only one the PetscInitialize(); > > I want also to use: > eigensolver eig1; > eigensolver eig2; > > I call only once PetscInitialize(); for each eig1 and eig 2. > > It is much better to call PetscInitialize only once, ever. There are lots of ways to set that up in your code. I would use a static variable and register a finalize routine with my main. But note that it is perfectly legitimate to have PetscInitialize() PetscFinalize() PetscInitialize() PetscFinalize() etc etc in your code. Though it is not recommended, it should work so long as YOU call MPI_Init () (if using MPI) before the first PetscInitialize() and MPI_Finalize() after the final PetscFinalize(). Barry Note that some of the petsc-developers still falsely believe they can only be called once. From jedbrown at mcs.anl.gov Wed Oct 3 19:08:21 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Oct 2012 19:08:21 -0500 Subject: [petsc-users] PetscFinalize() In-Reply-To: <205C27BA-1161-4EAC-81D3-EA9B889E577E@mcs.anl.gov> References: <205C27BA-1161-4EAC-81D3-EA9B889E577E@mcs.anl.gov> Message-ID: On Wed, Oct 3, 2012 at 6:53 PM, Barry Smith wrote: > Note that some of the petsc-developers still falsely believe they can only > be called once. I know it _can_ be called this way, but I still strongly recommend that people don't do this. So much so that I'd rather not reveal that its possible unless someone describes the extenuating circumstances. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 3 20:40:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Oct 2012 21:40:57 -0400 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506C9A99.60300@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> Message-ID: On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: > Hi Jed, > > I believe they are real cores. Anyway, I have attached the log summary for > the 12/24/48 cores. I re-run a smaller case because the large problem can't > run with 12cores. > > Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing you have 4 16-core chips, but please figure this out). The messages are logged in ScatterBegin, and the time is logged in ScatterEnd. From 24 to 48 cores the time is cut in half. If you were only communicating the boundary, this is completely backwards, so you are communicating a fair fraction of ALL the values in a subdomain. Figure out why your partition is so screwed up and this will go away. Matt > Yours sincerely, > > TAY wee-beng > > On 3/10/2012 5:59 PM, Jed Brown wrote: > > There is an inordinate amount of time being spent in VecScatterEnd(). That > sometimes indicates a very bad partition. Also, are your "48 cores" real > physical cores or just "logical cores" (look like cores to the operating > system, usually advertised as "threads" by the vendor, nothing like cores > in reality)? That can cause a huge load imbalance and very confusing > results as over-subscribed threads compete for shared resources. Step it > back to 24 threads and 12 threads, send log_summary for each. > > On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: > >> On 2/10/2012 2:43 PM, Jed Brown wrote: >> >> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >> >>> Hi, >>> >>> I have combined the momentum linear eqns involving x,y,z into 1 large >>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>> procs. The log_summary is given below. I have some questions: >>> >>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>> does it says there are 4 matrix, 12 vector etc? >>> >> >> They are part of preconditioning. Are you sure you're using Hypre for >> this? It looks like you are using bjacobi/ilu. >> >> >>> >>> 2. I'm looking at the stages which take the longest time. It seems that >>> MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high >>> ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are >>> these stages the reason why my code is not scaling well? What can I do to >>> improve it? >>> >> >> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >> MatLUFactorNumeric, and VecNorm+VecDot. >> >> The high VecAssembly time might be due to generating a lot of entries >> off-process? >> >> In any case, this looks like an _extremely_ slow network, perhaps it's >> misconfigured? >> >> >> My cluster is configured with 48 procs per node. I re-run the case, >> using only 48 procs, thus there's no need to pass over a 'slow' >> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >> eqn respectively. I have also separated the x,y,z component of the momentum >> eqn to 3 separate linear eqns to debug the problem. >> >> Results show that stage "momentum_z" is taking a lot of time. I wonder if >> it has to do with the fact that I am partitioning my grids in the z >> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >> >> I wonder why a lot of entries are generated off-process. >> >> I create my RHS vector using: >> >> *call >> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >> * >> >> where ijk_xyz_sta and ijk_xyz_end are obtained from >> >> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >> >> I then insert the values into the vector using: >> >> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >> ijk_xyz_end) , INSERT_VALUES , ierr)* >> >> What should I do to correct the problem? >> >> Thanks >> >> >> >> >>> >>> Btw, I insert matrix using: >>> >>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>> ** >>> ** II = ijk - 1** !Fortran shift to 0-based** >>> ** ** >>> ** call >>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>> ** >>> ** >>> **end do* >>> >>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>> >>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>> >>> semi_mat_xyz has the corresponding values. >>> >>> and I insert vectors using: >>> >>> call >>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>> >>> Thanks! >>> >>> * >>> * >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>> >>> You can measure the time spent in Hypre via PCApply and PCSetUp, but you >>> can't get finer grained integrated profiling because it was not set up that >>> way. >>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>> >>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>> >>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>> >>>>> 1. if I'm solving 3 linear equations using ksp, is the result given in >>>>> the log summary the total of the 3 linear eqns' performance? How can I get >>>>> the performance for each individual eqn? >>>>> >>>> >>>> Use logging stages: >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>> >>>> >>>>> 2. If I run my code for 10 time steps, does the log summary gives the >>>>> total or avg performance/ratio? >>>>> >>>> >>>> Total. >>>> >>>> >>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) >>>>> to solve my Cartesian's grid CFD poisson eqn. Is there any way I can use >>>>> PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru >>>>> PETSc, can I get its performance? >>>> >>>> >>>> If you mean flops, only if you count them yourself and tell PETSc >>>> using >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>> >>>> This is the disadvantage of using packages that do not properly >>>> monitor things :) >>>> >>>> Matt >>>> >>>> >>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>> evaluating its performance, beside using PetscLogFlops? >>>> >>>> >>>>> -- >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> >>>> >>> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Wed Oct 3 22:25:13 2012 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 4 Oct 2012 11:25:13 +0800 (CST) Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> Message-ID: <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> Only implemented this strategy in TS is not enough, i.e. DC sweep (each DC point requires nonlinear iteration) may also need it. Will you implement it into the SNES and send me the patch? For example, a flag -3 to SNESSetLagPreconditioner. At the first ineration of each SENSSolve call, the PC will be built and never change in the same SENSSolve iteration. I'd like to try it asap since this strategy improves simulation speed significantly. Gong Ding I've been meaning to implement this for a while. You can put custom SNES control into a pre-stage or pre-step hook, but this particular case should, in my opinion, be provided as a TS option. On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: Hi, I am doing a nonlinear transient simulation. For each time step, a SNESSolve is called. I want to do a LU factorization at the first Newton iteration of EACH time step, and for the following Newton steps (in the same time step), no factorization any more. For the next time step, the LU factorization should be done again. SNESSetLagPreconditioner(-1) seems just skips all the following factorization, even in the different time step. How can I do? Gong Ding -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 3 22:28:17 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 3 Oct 2012 22:28:17 -0500 Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> Message-ID: If you are calling SNESSolve(), just call SNESSetLagPreconditioner() again before each solve. On Wed, Oct 3, 2012 at 10:25 PM, Gong Ding wrote: > Only implemented this strategy in TS is not enough, i.e. DC sweep (each DC > point requires nonlinear iteration) may also need it. > Will you implement it into the SNES and send me the patch? > For example, a flag -3 to SNESSetLagPreconditioner. > At the first ineration of each SENSSolve call, the PC will be built and > never change in the same SENSSolve iteration. > I'd like to try it asap since this strategy improves simulation speed > significantly. > > Gong Ding > > > > > > I've been meaning to implement this for a while. You can put custom SNES > control into a pre-stage or pre-step hook, but this particular case should, > in my opinion, be provided as a TS option. > > On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: > >> Hi, >> I am doing a nonlinear transient simulation. For each time step, a >> SNESSolve is called. >> I want to do a LU factorization at the first Newton iteration of EACH >> time step, and for the following Newton steps (in the same time step), no >> factorization any more. For the next time step, the LU factorization should >> be done again. >> >> >> SNESSetLagPreconditioner(-1) seems just skips all the following >> factorization, even in the different time step. >> How can I do? >> >> >> Gong Ding >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Wed Oct 3 23:07:51 2012 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 4 Oct 2012 12:07:51 +0800 (CST) Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> Message-ID: <11059273.348501349323671676.JavaMail.coremail@mail.ustc.edu> call SNESSetLagPreconditioner before SNESSolve is useless. It seems SNESSetLagPreconditioner does not reset the internal information. If you are calling SNESSolve(), just call SNESSetLagPreconditioner() again before each solve. On Wed, Oct 3, 2012 at 10:25 PM, Gong Ding wrote: Only implemented this strategy in TS is not enough, i.e. DC sweep (each DC point requires nonlinear iteration) may also need it. Will you implement it into the SNES and send me the patch? For example, a flag -3 to SNESSetLagPreconditioner. At the first ineration of each SENSSolve call, the PC will be built and never change in the same SENSSolve iteration. I'd like to try it asap since this strategy improves simulation speed significantly. Gong Ding I've been meaning to implement this for a while. You can put custom SNES control into a pre-stage or pre-step hook, but this particular case should, in my opinion, be provided as a TS option. On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: Hi, I am doing a nonlinear transient simulation. For each time step, a SNESSolve is called. I want to do a LU factorization at the first Newton iteration of EACH time step, and for the following Newton steps (in the same time step), no factorization any more. For the next time step, the LU factorization should be done again. SNESSetLagPreconditioner(-1) seems just skips all the following factorization, even in the different time step. How can I do? Gong Ding -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 3 23:10:21 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Oct 2012 00:10:21 -0400 Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: <11059273.348501349323671676.JavaMail.coremail@mail.ustc.edu> References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> <11059273.348501349323671676.JavaMail.coremail@mail.ustc.edu> Message-ID: On Thu, Oct 4, 2012 at 12:07 AM, Gong Ding wrote: > call SNESSetLagPreconditioner before SNESSolve is useless. > It seems SNESSetLagPreconditioner does not reset the internal information. > > I am looking directly at the code: http://www.mcs.anl.gov/petsc/petsc-dev/src/snes/interface/snes.c.html#SNESSetLagPreconditioner It definitely changes the "internal information". Matt > If you are calling SNESSolve(), just call SNESSetLagPreconditioner() again > before each solve. > > > On Wed, Oct 3, 2012 at 10:25 PM, Gong Ding wrote: > >> Only implemented this strategy in TS is not enough, i.e. DC sweep (each >> DC point requires nonlinear iteration) may also need it. >> Will you implement it into the SNES and send me the patch? >> For example, a flag -3 to SNESSetLagPreconditioner. >> At the first ineration of each SENSSolve call, the PC will be built and >> never change in the same SENSSolve iteration. >> I'd like to try it asap since this strategy improves simulation speed >> significantly. >> >> Gong Ding >> >> >> >> >> >> I've been meaning to implement this for a while. You can put custom SNES >> control into a pre-stage or pre-step hook, but this particular case should, >> in my opinion, be provided as a TS option. >> >> On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: >> >>> Hi, >>> I am doing a nonlinear transient simulation. For each time step, a >>> SNESSolve is called. >>> I want to do a LU factorization at the first Newton iteration of EACH >>> time step, and for the following Newton steps (in the same time step), no >>> factorization any more. For the next time step, the LU factorization should >>> be done again. >>> >>> >>> SNESSetLagPreconditioner(-1) seems just skips all the following >>> factorization, even in the different time step. >>> How can I do? >>> >>> >>> Gong Ding >>> >>> >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdiso at ustc.edu Thu Oct 4 00:00:49 2012 From: gdiso at ustc.edu (Gong Ding) Date: Thu, 4 Oct 2012 13:00:49 +0800 (CST) Subject: [petsc-users] More about SNESSetLagPreconditioner In-Reply-To: References: <2117518.348241349276456947.JavaMail.coremail@mail.ustc.edu> <19904100.348461349321113610.JavaMail.coremail@mail.ustc.edu> <11059273.348501349323671676.JavaMail.coremail@mail.ustc.edu> Message-ID: <25850462.348521349326849612.JavaMail.coremail@mail.ustc.edu> Thank you for giving the entry of source code. I think I can patch it for my purpose. On Thu, Oct 4, 2012 at 12:07 AM, Gong Ding wrote: call SNESSetLagPreconditioner before SNESSolve is useless. It seems SNESSetLagPreconditioner does not reset the internal information. I am looking directly at the code: http://www.mcs.anl.gov/petsc/petsc-dev/src/snes/interface/snes.c.html#SNESSetLagPreconditioner It definitely changes the "internal information". Matt If you are calling SNESSolve(), just call SNESSetLagPreconditioner() again before each solve. On Wed, Oct 3, 2012 at 10:25 PM, Gong Ding wrote: Only implemented this strategy in TS is not enough, i.e. DC sweep (each DC point requires nonlinear iteration) may also need it. Will you implement it into the SNES and send me the patch? For example, a flag -3 to SNESSetLagPreconditioner. At the first ineration of each SENSSolve call, the PC will be built and never change in the same SENSSolve iteration. I'd like to try it asap since this strategy improves simulation speed significantly. Gong Ding I've been meaning to implement this for a while. You can put custom SNES control into a pre-stage or pre-step hook, but this particular case should, in my opinion, be provided as a TS option. On Wed, Oct 3, 2012 at 11:00 AM, Gong Ding wrote: Hi, I am doing a nonlinear transient simulation. For each time step, a SNESSolve is called. I want to do a LU factorization at the first Newton iteration of EACH time step, and for the following Newton steps (in the same time step), no factorization any more. For the next time step, the LU factorization should be done again. SNESSetLagPreconditioner(-1) seems just skips all the following factorization, even in the different time step. How can I do? Gong Ding -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Oct 4 10:01:52 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 04 Oct 2012 17:01:52 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> Message-ID: <506DA4E0.5040503@gmail.com> On 4/10/2012 3:40 AM, Matthew Knepley wrote: > On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng > wrote: > > Hi Jed, > > I believe they are real cores. Anyway, I have attached the log > summary for the 12/24/48 cores. I re-run a smaller case because > the large problem can't run with 12cores. > > > Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing > you have 4 16-core chips, but please figure this out). > The messages are logged in ScatterBegin, and the time is logged in > ScatterEnd. From 24 to 48 cores the time is cut in half. > If you were only communicating the boundary, this is completely > backwards, so you are communicating a fair fraction of ALL > the values in a subdomain. Figure out why your partition is so screwed > up and this will go away. What do you mean by "If you were only communicating the boundary, this is completely backwards, so you are communicating a fair fraction of ALL the values in a subdomain"? I partition my domain in the z direction, as shown in the attached pic. The circled region is where the airfoils are. I'm using an immersed boundary method (IBM) code so the grid is all Cartesian. I created my Z matrix using: call MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) where ijk_sta / ijk_end are the starting/ending global indices of the row. 7 is because the star-stencil is used in 3D. I create my RHS vector using: /call VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ The values for the matrix and vector were calculated before PETSc logging so they don't come into play. They are also done in a similar fashion for matrix x and y. I still can't get it why solving the z momentum eqn takes so much time. Which portion should I focus on? Tks! > > Matt > > Yours sincerely, > > TAY wee-beng > > On 3/10/2012 5:59 PM, Jed Brown wrote: >> There is an inordinate amount of time being spent in >> VecScatterEnd(). That sometimes indicates a very bad partition. >> Also, are your "48 cores" real physical cores or just "logical >> cores" (look like cores to the operating system, usually >> advertised as "threads" by the vendor, nothing like cores in >> reality)? That can cause a huge load imbalance and very confusing >> results as over-subscribed threads compete for shared resources. >> Step it back to 24 threads and 12 threads, send log_summary for each. >> >> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng > > wrote: >> >> On 2/10/2012 2:43 PM, Jed Brown wrote: >>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng >>> > wrote: >>> >>> Hi, >>> >>> I have combined the momentum linear eqns involving x,y,z >>> into 1 large matrix. The Poisson eqn is solved using >>> HYPRE strcut format so it's not included. I run the code >>> for 50 timesteps (hence 50 kspsolve) using 96 procs. The >>> log_summary is given below. I have some questions: >>> >>> 1. After combining the matrix, I should have only 1 >>> PETSc matrix. Why does it says there are 4 matrix, 12 >>> vector etc? >>> >>> >>> They are part of preconditioning. Are you sure you're using >>> Hypre for this? It looks like you are using bjacobi/ilu. >>> >>> >>> 2. I'm looking at the stages which take the longest >>> time. It seems that MatAssemblyBegin, VecNorm, >>> VecAssemblyBegin, VecScatterEnd have very high ratios. >>> The ratios of some others are also not too good (~ 1.6 - >>> 2). So are these stages the reason why my code is not >>> scaling well? What can I do to improve it? >>> >>> >>> 3/4 of the solve time is evenly balanced between MatMult, >>> MatSolve, MatLUFactorNumeric, and VecNorm+VecDot. >>> >>> The high VecAssembly time might be due to generating a lot >>> of entries off-process? >>> >>> In any case, this looks like an _extremely_ slow network, >>> perhaps it's misconfigured? >> >> My cluster is configured with 48 procs per node. I re-run the >> case, using only 48 procs, thus there's no need to pass over >> a 'slow' interconnect. I'm now also using GAMG and BCGS for >> the poisson and momentum eqn respectively. I have also >> separated the x,y,z component of the momentum eqn to 3 >> separate linear eqns to debug the problem. >> >> Results show that stage "momentum_z" is taking a lot of time. >> I wonder if it has to do with the fact that I am partitioning >> my grids in the z direction. VecScatterEnd, MatMult are >> taking a lot of time. VecNormalize, VecScatterEnd, VecNorm, >> VecAssemblyBegin 's ratio are also not good. >> >> I wonder why a lot of entries are generated off-process. >> >> I create my RHS vector using: >> >> /call >> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ >> >> where ijk_xyz_sta and ijk_xyz_end are obtained from >> >> /call >> MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)/ >> >> I then insert the values into the vector using: >> >> /call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta >> + 1 : ijk_xyz_end) , INSERT_VALUES , ierr)/ >> >> What should I do to correct the problem? >> >> Thanks >> >> >>> >>> Btw, I insert matrix using: >>> >>> /do ijk=ijk_xyz_sta+1,ijk_xyz_end// >>> // >>> // II = ijk - 1//!Fortran shift to 0-based// >>> //// >>> // call >>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// >>> // >>> //end do/ >>> >>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>> >>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>> >>> semi_mat_xyz has the corresponding values. >>> >>> and I insert vectors using: >>> >>> call >>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>> >>> Thanks! >>> >>> / >>> / >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>> >>>> You can measure the time spent in Hypre via PCApply and >>>> PCSetUp, but you can't get finer grained integrated >>>> profiling because it was not set up that way. >>>> >>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" >>>> > wrote: >>>> >>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng >>>>> > wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm doing a log summary for my 3d cfd code. I >>>>> have some questions: >>>>> >>>>> 1. if I'm solving 3 linear equations using >>>>> ksp, is the result given in the log summary >>>>> the total of the 3 linear eqns' performance? >>>>> How can I get the performance for each >>>>> individual eqn? >>>>> >>>>> >>>>> Use logging stages: >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>> >>>>> 2. If I run my code for 10 time steps, does >>>>> the log summary gives the total or avg >>>>> performance/ratio? >>>>> >>>>> >>>>> Total. >>>>> >>>>> 3. Besides PETSc, I'm also using HYPRE's >>>>> native geometric MG (Struct) to solve my >>>>> Cartesian's grid CFD poisson eqn. Is there any >>>>> way I can use PETSc's log summary to get >>>>> HYPRE's performance? If I use boomerAMG thru >>>>> PETSc, can I get its performance? >>>>> >>>>> >>>>> If you mean flops, only if you count them yourself >>>>> and tell PETSc using >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>> >>>>> This is the disadvantage of using packages that do >>>>> not properly monitor things :) >>>>> >>>>> Matt >>>> So u mean if I use boomerAMG thru PETSc, there is >>>> no proper way of evaluating its performance, beside >>>> using PetscLogFlops? >>>>> >>>>> >>>>> -- >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before >>>>> they begin their experiments is infinitely more >>>>> interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>> >>> >>> >> >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 3d_grid.jpg Type: image/jpeg Size: 179125 bytes Desc: not available URL: From knepley at gmail.com Thu Oct 4 10:11:11 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Oct 2012 11:11:11 -0400 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506DA4E0.5040503@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> Message-ID: On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng wrote: > On 4/10/2012 3:40 AM, Matthew Knepley wrote: > > On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: > >> Hi Jed, >> >> I believe they are real cores. Anyway, I have attached the log summary >> for the 12/24/48 cores. I re-run a smaller case because the large problem >> can't run with 12cores. >> > > Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing you > have 4 16-core chips, but please figure this out). > The messages are logged in ScatterBegin, and the time is logged in > ScatterEnd. From 24 to 48 cores the time is cut in half. > If you were only communicating the boundary, this is completely backwards, > so you are communicating a fair fraction of ALL > the values in a subdomain. Figure out why your partition is so screwed up > and this will go away. > > > What do you mean by "If you were only communicating the boundary, this is > completely backwards, so you are communicating a fair fraction of ALL the > values in a subdomain"? > If you have 48 partitions instead of 24, you have a larger interface, so AssemblyEnd() should take slightly longer. However, your AssemblyEnd() takes HALF the time, which means its communicating much fewer values, which means you are not sending interface values, you are sending interior values, since the interior shrinks when you have more partitions. What this probably means is that your assembly routines are screwed up, and sending data all over the place. Matt > I partition my domain in the z direction, as shown in the attached pic. > The circled region is where the airfoils are. I'm using an immersed > boundary method (IBM) code so the grid is all Cartesian. > > I created my Z matrix using: > > call > MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) > > where ijk_sta / ijk_end are the starting/ending global indices of the row. > > 7 is because the star-stencil is used in 3D. > > I create my RHS vector using: > > *call > VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) > * > > The values for the matrix and vector were calculated before PETSc logging > so they don't come into play. > > They are also done in a similar fashion for matrix x and y. I still can't > get it why solving the z momentum eqn takes so much time. Which portion > should I focus on? > > Tks! > > > Matt > > >> Yours sincerely, >> >> TAY wee-beng >> >> On 3/10/2012 5:59 PM, Jed Brown wrote: >> >> There is an inordinate amount of time being spent in VecScatterEnd(). >> That sometimes indicates a very bad partition. Also, are your "48 cores" >> real physical cores or just "logical cores" (look like cores to the >> operating system, usually advertised as "threads" by the vendor, nothing >> like cores in reality)? That can cause a huge load imbalance and very >> confusing results as over-subscribed threads compete for shared resources. >> Step it back to 24 threads and 12 threads, send log_summary for each. >> >> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: >> >>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>> >>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >>> >>>> Hi, >>>> >>>> I have combined the momentum linear eqns involving x,y,z into 1 large >>>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>>> procs. The log_summary is given below. I have some questions: >>>> >>>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>>> does it says there are 4 matrix, 12 vector etc? >>>> >>> >>> They are part of preconditioning. Are you sure you're using Hypre for >>> this? It looks like you are using bjacobi/ilu. >>> >>> >>>> >>>> 2. I'm looking at the stages which take the longest time. It seems that >>>> MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high >>>> ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are >>>> these stages the reason why my code is not scaling well? What can I do to >>>> improve it? >>>> >>> >>> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >>> MatLUFactorNumeric, and VecNorm+VecDot. >>> >>> The high VecAssembly time might be due to generating a lot of entries >>> off-process? >>> >>> In any case, this looks like an _extremely_ slow network, perhaps it's >>> misconfigured? >>> >>> >>> My cluster is configured with 48 procs per node. I re-run the case, >>> using only 48 procs, thus there's no need to pass over a 'slow' >>> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >>> eqn respectively. I have also separated the x,y,z component of the momentum >>> eqn to 3 separate linear eqns to debug the problem. >>> >>> Results show that stage "momentum_z" is taking a lot of time. I wonder >>> if it has to do with the fact that I am partitioning my grids in the z >>> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >>> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >>> >>> I wonder why a lot of entries are generated off-process. >>> >>> I create my RHS vector using: >>> >>> *call >>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>> * >>> >>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>> >>> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >>> >>> I then insert the values into the vector using: >>> >>> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >>> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >>> ijk_xyz_end) , INSERT_VALUES , ierr)* >>> >>> What should I do to correct the problem? >>> >>> Thanks >>> >>> >>> >>> >>>> >>>> Btw, I insert matrix using: >>>> >>>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>>> ** >>>> ** II = ijk - 1** !Fortran shift to 0-based** >>>> ** ** >>>> ** call >>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>>> ** >>>> ** >>>> **end do* >>>> >>>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>>> >>>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>>> >>>> semi_mat_xyz has the corresponding values. >>>> >>>> and I insert vectors using: >>>> >>>> call >>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>> >>>> Thanks! >>>> >>>> * >>>> * >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>> >>>> You can measure the time spent in Hypre via PCApply and PCSetUp, but >>>> you can't get finer grained integrated profiling because it was not set up >>>> that way. >>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>>> >>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>> >>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>>> >>>>>> 1. if I'm solving 3 linear equations using ksp, is the result given >>>>>> in the log summary the total of the 3 linear eqns' performance? How can I >>>>>> get the performance for each individual eqn? >>>>>> >>>>> >>>>> Use logging stages: >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>> >>>>> >>>>>> 2. If I run my code for 10 time steps, does the log summary gives the >>>>>> total or avg performance/ratio? >>>>>> >>>>> >>>>> Total. >>>>> >>>>> >>>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) >>>>>> to solve my Cartesian's grid CFD poisson eqn. Is there any way I can use >>>>>> PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru >>>>>> PETSc, can I get its performance? >>>>> >>>>> >>>>> If you mean flops, only if you count them yourself and tell PETSc >>>>> using >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>> >>>>> This is the disadvantage of using packages that do not properly >>>>> monitor things :) >>>>> >>>>> Matt >>>>> >>>>> >>>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>>> evaluating its performance, beside using PetscLogFlops? >>>>> >>>>> >>>>>> -- >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Oct 4 14:16:25 2012 From: zonexo at gmail.com (Wee-Beng Tay) Date: Thu, 4 Oct 2012 21:16:25 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> Message-ID: On 4/10/2012 5:11 PM, Matthew Knepley wrote: On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng wrote: > On 4/10/2012 3:40 AM, Matthew Knepley wrote: > > On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: > >> Hi Jed, >> >> I believe they are real cores. Anyway, I have attached the log summary >> for the 12/24/48 cores. I re-run a smaller case because the large problem >> can't run with 12cores. >> > > Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing you > have 4 16-core chips, but please figure this out). > The messages are logged in ScatterBegin, and the time is logged in > ScatterEnd. From 24 to 48 cores the time is cut in half. > If you were only communicating the boundary, this is completely backwards, > so you are communicating a fair fraction of ALL > the values in a subdomain. Figure out why your partition is so screwed up > and this will go away. > > > What do you mean by "If you were only communicating the boundary, this is > completely backwards, so you are communicating a fair fraction of ALL the > values in a subdomain"? > If you have 48 partitions instead of 24, you have a larger interface, so AssemblyEnd() should take slightly longer. However, your AssemblyEnd() takes HALF the time, which means its communicating much fewer values, which means you are not sending interface values, you are sending interior values, since the interior shrinks when you have more partitions. What this probably means is that your assembly routines are screwed up, and sending data all over the place. Ok I got it now. Looking at the AssemblyEnd time, 12 procs MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 24 procs MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 48 procs MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd time increases with procs, does it mean that there is nothing wrong with it? On the other hand, MatAssemblyEnd time decreases with procs. So that's where the problem lies, is that so? I'm still scanning my code but haven't found the error yet. It seems strange because I inserted the matrix and vector exactly the same way for x,y,z. The u,v,w are also allocated with the same indices. Shouldn't the error be the same for x, y and z too? Trying to get some hints as to where else I need to look in my code... Tks Matt > I partition my domain in the z direction, as shown in the attached pic. > The circled region is where the airfoils are. I'm using an immersed > boundary method (IBM) code so the grid is all Cartesian. > > I created my Z matrix using: > > call > MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) > > where ijk_sta / ijk_end are the starting/ending global indices of the row. > > 7 is because the star-stencil is used in 3D. > > I create my RHS vector using: > > *call > VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) > * > > The values for the matrix and vector were calculated before PETSc logging > so they don't come into play. > > They are also done in a similar fashion for matrix x and y. I still can't > get it why solving the z momentum eqn takes so much time. Which portion > should I focus on? > > Tks! > > > Matt > > >> Yours sincerely, >> >> TAY wee-beng >> >> On 3/10/2012 5:59 PM, Jed Brown wrote: >> >> There is an inordinate amount of time being spent in VecScatterEnd(). >> That sometimes indicates a very bad partition. Also, are your "48 cores" >> real physical cores or just "logical cores" (look like cores to the >> operating system, usually advertised as "threads" by the vendor, nothing >> like cores in reality)? That can cause a huge load imbalance and very >> confusing results as over-subscribed threads compete for shared resources. >> Step it back to 24 threads and 12 threads, send log_summary for each. >> >> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: >> >>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>> >>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >>> >>>> Hi, >>>> >>>> I have combined the momentum linear eqns involving x,y,z into 1 large >>>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>>> procs. The log_summary is given below. I have some questions: >>>> >>>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>>> does it says there are 4 matrix, 12 vector etc? >>>> >>> >>> They are part of preconditioning. Are you sure you're using Hypre for >>> this? It looks like you are using bjacobi/ilu. >>> >>> >>>> >>>> 2. I'm looking at the stages which take the longest time. It seems that >>>> MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very high >>>> ratios. The ratios of some others are also not too good (~ 1.6 - 2). So are >>>> these stages the reason why my code is not scaling well? What can I do to >>>> improve it? >>>> >>> >>> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >>> MatLUFactorNumeric, and VecNorm+VecDot. >>> >>> The high VecAssembly time might be due to generating a lot of entries >>> off-process? >>> >>> In any case, this looks like an _extremely_ slow network, perhaps it's >>> misconfigured? >>> >>> >>> My cluster is configured with 48 procs per node. I re-run the case, >>> using only 48 procs, thus there's no need to pass over a 'slow' >>> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >>> eqn respectively. I have also separated the x,y,z component of the momentum >>> eqn to 3 separate linear eqns to debug the problem. >>> >>> Results show that stage "momentum_z" is taking a lot of time. I wonder >>> if it has to do with the fact that I am partitioning my grids in the z >>> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >>> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >>> >>> I wonder why a lot of entries are generated off-process. >>> >>> I create my RHS vector using: >>> >>> *call >>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>> * >>> >>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>> >>> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >>> >>> I then insert the values into the vector using: >>> >>> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >>> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >>> ijk_xyz_end) , INSERT_VALUES , ierr)* >>> >>> What should I do to correct the problem? >>> >>> Thanks >>> >>> >>> >>> >>>> >>>> Btw, I insert matrix using: >>>> >>>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>>> ** >>>> ** II = ijk - 1** !Fortran shift to 0-based** >>>> ** ** >>>> ** call >>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>>> ** >>>> ** >>>> **end do* >>>> >>>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>>> >>>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>>> >>>> semi_mat_xyz has the corresponding values. >>>> >>>> and I insert vectors using: >>>> >>>> call >>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>> >>>> Thanks! >>>> >>>> * >>>> * >>>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>> >>>> You can measure the time spent in Hypre via PCApply and PCSetUp, but >>>> you can't get finer grained integrated profiling because it was not set up >>>> that way. >>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>>> >>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>> >>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>>> >>>>>> 1. if I'm solving 3 linear equations using ksp, is the result given >>>>>> in the log summary the total of the 3 linear eqns' performance? How can I >>>>>> get the performance for each individual eqn? >>>>>> >>>>> >>>>> Use logging stages: >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>> >>>>> >>>>>> 2. If I run my code for 10 time steps, does the log summary gives the >>>>>> total or avg performance/ratio? >>>>>> >>>>> >>>>> Total. >>>>> >>>>> >>>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG (Struct) >>>>>> to solve my Cartesian's grid CFD poisson eqn. Is there any way I can use >>>>>> PETSc's log summary to get HYPRE's performance? If I use boomerAMG thru >>>>>> PETSc, can I get its performance? >>>>> >>>>> >>>>> If you mean flops, only if you count them yourself and tell PETSc >>>>> using >>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>> >>>>> This is the disadvantage of using packages that do not properly >>>>> monitor things :) >>>>> >>>>> Matt >>>>> >>>>> >>>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>>> evaluating its performance, beside using PetscLogFlops? >>>>> >>>>> >>>>>> -- >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> >>>>> >>>> >>> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Oct 4 14:21:22 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 4 Oct 2012 14:21:22 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> Message-ID: Can you send a picture of what your domain looks like and what shape the part owned by a given processor looks like? Best would be to write out the mesh with a variable marking the rank owning each vertex, then do a color plot in Paraview or whatever you use to show the partition. VecScatterBegin/End is taking much more time than these, and really a pretty unreasonable amount of time in general. On Thu, Oct 4, 2012 at 2:16 PM, Wee-Beng Tay wrote: > On 4/10/2012 5:11 PM, Matthew Knepley wrote: > > On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng wrote: > >> On 4/10/2012 3:40 AM, Matthew Knepley wrote: >> >> On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: >> >>> Hi Jed, >>> >>> I believe they are real cores. Anyway, I have attached the log summary >>> for the 12/24/48 cores. I re-run a smaller case because the large problem >>> can't run with 12cores. >>> >> >> Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing >> you have 4 16-core chips, but please figure this out). >> The messages are logged in ScatterBegin, and the time is logged in >> ScatterEnd. From 24 to 48 cores the time is cut in half. >> If you were only communicating the boundary, this is completely >> backwards, so you are communicating a fair fraction of ALL >> the values in a subdomain. Figure out why your partition is so screwed up >> and this will go away. >> >> >> What do you mean by "If you were only communicating the boundary, this is >> completely backwards, so you are communicating a fair fraction of ALL the >> values in a subdomain"? >> > > If you have 48 partitions instead of 24, you have a larger interface, so > AssemblyEnd() should take > slightly longer. However, your AssemblyEnd() takes HALF the time, which > means its communicating > much fewer values, which means you are not sending interface values, you > are sending interior values, > since the interior shrinks when you have more partitions. > > What this probably means is that your assembly routines are screwed up, > and sending data all over the place. > > Ok I got it now. Looking at the AssemblyEnd time, > > 12 procs > > MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 24 procs > > MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 48 procs > > MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 > > > VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd time increases with procs, does it mean that there is > nothing wrong with it? > > On the other hand, MatAssemblyEnd time decreases with procs. So that's > where the problem lies, is that so? > > I'm still scanning my code but haven't found the error yet. It seems > strange because I inserted the matrix and vector exactly the same way for > x,y,z. The u,v,w are also allocated with the same indices. Shouldn't the > error be the same for x, y and z too? > > Trying to get some hints as to where else I need to look in my code... > > Tks > > > > > > > > Matt > > >> I partition my domain in the z direction, as shown in the attached pic. >> The circled region is where the airfoils are. I'm using an immersed >> boundary method (IBM) code so the grid is all Cartesian. >> >> I created my Z matrix using: >> >> call >> MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) >> >> where ijk_sta / ijk_end are the starting/ending global indices of the row. >> >> 7 is because the star-stencil is used in 3D. >> >> I create my RHS vector using: >> >> *call >> VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >> * >> >> The values for the matrix and vector were calculated before PETSc logging >> so they don't come into play. >> >> They are also done in a similar fashion for matrix x and y. I still can't >> get it why solving the z momentum eqn takes so much time. Which portion >> should I focus on? >> >> Tks! >> >> >> Matt >> >> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 3/10/2012 5:59 PM, Jed Brown wrote: >>> >>> There is an inordinate amount of time being spent in VecScatterEnd(). >>> That sometimes indicates a very bad partition. Also, are your "48 cores" >>> real physical cores or just "logical cores" (look like cores to the >>> operating system, usually advertised as "threads" by the vendor, nothing >>> like cores in reality)? That can cause a huge load imbalance and very >>> confusing results as over-subscribed threads compete for shared resources. >>> Step it back to 24 threads and 12 threads, send log_summary for each. >>> >>> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: >>> >>>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>>> >>>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >>>> >>>>> Hi, >>>>> >>>>> I have combined the momentum linear eqns involving x,y,z into 1 large >>>>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>>>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>>>> procs. The log_summary is given below. I have some questions: >>>>> >>>>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>>>> does it says there are 4 matrix, 12 vector etc? >>>>> >>>> >>>> They are part of preconditioning. Are you sure you're using Hypre for >>>> this? It looks like you are using bjacobi/ilu. >>>> >>>> >>>>> >>>>> 2. I'm looking at the stages which take the longest time. It seems >>>>> that MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very >>>>> high ratios. The ratios of some others are also not too good (~ 1.6 - 2). >>>>> So are these stages the reason why my code is not scaling well? What can I >>>>> do to improve it? >>>>> >>>> >>>> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >>>> MatLUFactorNumeric, and VecNorm+VecDot. >>>> >>>> The high VecAssembly time might be due to generating a lot of entries >>>> off-process? >>>> >>>> In any case, this looks like an _extremely_ slow network, perhaps >>>> it's misconfigured? >>>> >>>> >>>> My cluster is configured with 48 procs per node. I re-run the case, >>>> using only 48 procs, thus there's no need to pass over a 'slow' >>>> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >>>> eqn respectively. I have also separated the x,y,z component of the momentum >>>> eqn to 3 separate linear eqns to debug the problem. >>>> >>>> Results show that stage "momentum_z" is taking a lot of time. I wonder >>>> if it has to do with the fact that I am partitioning my grids in the z >>>> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >>>> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >>>> >>>> I wonder why a lot of entries are generated off-process. >>>> >>>> I create my RHS vector using: >>>> >>>> *call >>>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>>> * >>>> >>>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>>> >>>> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >>>> >>>> I then insert the values into the vector using: >>>> >>>> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >>>> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >>>> ijk_xyz_end) , INSERT_VALUES , ierr)* >>>> >>>> What should I do to correct the problem? >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>>> >>>>> Btw, I insert matrix using: >>>>> >>>>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>>>> ** >>>>> ** II = ijk - 1** !Fortran shift to 0-based** >>>>> ** ** >>>>> ** call >>>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>>>> ** >>>>> ** >>>>> **end do* >>>>> >>>>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>>>> >>>>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>>>> >>>>> semi_mat_xyz has the corresponding values. >>>>> >>>>> and I insert vectors using: >>>>> >>>>> call >>>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>>> >>>>> Thanks! >>>>> >>>>> * >>>>> * >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>>> >>>>> You can measure the time spent in Hypre via PCApply and PCSetUp, but >>>>> you can't get finer grained integrated profiling because it was not set up >>>>> that way. >>>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>>>> >>>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>>> >>>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>>>> >>>>>>> 1. if I'm solving 3 linear equations using ksp, is the result given >>>>>>> in the log summary the total of the 3 linear eqns' performance? How can I >>>>>>> get the performance for each individual eqn? >>>>>>> >>>>>> >>>>>> Use logging stages: >>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>>> >>>>>> >>>>>>> 2. If I run my code for 10 time steps, does the log summary gives >>>>>>> the total or avg performance/ratio? >>>>>>> >>>>>> >>>>>> Total. >>>>>> >>>>>> >>>>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG >>>>>>> (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there any way I >>>>>>> can use PETSc's log summary to get HYPRE's performance? If I use boomerAMG >>>>>>> thru PETSc, can I get its performance? >>>>>> >>>>>> >>>>>> If you mean flops, only if you count them yourself and tell PETSc >>>>>> using >>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>>> >>>>>> This is the disadvantage of using packages that do not properly >>>>>> monitor things :) >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>>>> evaluating its performance, beside using PetscLogFlops? >>>>>> >>>>>> >>>>>>> -- >>>>>>> Yours sincerely, >>>>>>> >>>>>>> TAY wee-beng >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 4 14:49:11 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Oct 2012 15:49:11 -0400 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> Message-ID: On Thu, Oct 4, 2012 at 3:16 PM, Wee-Beng Tay wrote: > On 4/10/2012 5:11 PM, Matthew Knepley wrote: > > On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng wrote: > >> On 4/10/2012 3:40 AM, Matthew Knepley wrote: >> >> On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: >> >>> Hi Jed, >>> >>> I believe they are real cores. Anyway, I have attached the log summary >>> for the 12/24/48 cores. I re-run a smaller case because the large problem >>> can't run with 12cores. >>> >> >> Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing >> you have 4 16-core chips, but please figure this out). >> The messages are logged in ScatterBegin, and the time is logged in >> ScatterEnd. From 24 to 48 cores the time is cut in half. >> If you were only communicating the boundary, this is completely >> backwards, so you are communicating a fair fraction of ALL >> the values in a subdomain. Figure out why your partition is so screwed up >> and this will go away. >> >> >> What do you mean by "If you were only communicating the boundary, this is >> completely backwards, so you are communicating a fair fraction of ALL the >> values in a subdomain"? >> > > If you have 48 partitions instead of 24, you have a larger interface, so > AssemblyEnd() should take > slightly longer. However, your AssemblyEnd() takes HALF the time, which > means its communicating > much fewer values, which means you are not sending interface values, you > are sending interior values, > since the interior shrinks when you have more partitions. > > What this probably means is that your assembly routines are screwed up, > and sending data all over the place. > > Ok I got it now. Looking at the AssemblyEnd time, > No no no no no. You are looking at completely wrong numbers. You MUST look at the Momentum-Z stage. Matt > 12 procs > > MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 24 procs > > MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 48 procs > > MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 > 8.0e+00 0 0 0 0 0 0 0 0 0 0 > > > VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd time increases with procs, does it mean that there is > nothing wrong with it? > > On the other hand, MatAssemblyEnd time decreases with procs. So that's > where the problem lies, is that so? > > I'm still scanning my code but haven't found the error yet. It seems > strange because I inserted the matrix and vector exactly the same way for > x,y,z. The u,v,w are also allocated with the same indices. Shouldn't the > error be the same for x, y and z too? > > Trying to get some hints as to where else I need to look in my code... > > Tks > > > > > > > Matt > > >> I partition my domain in the z direction, as shown in the attached pic. >> The circled region is where the airfoils are. I'm using an immersed >> boundary method (IBM) code so the grid is all Cartesian. >> >> I created my Z matrix using: >> >> call >> MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) >> >> where ijk_sta / ijk_end are the starting/ending global indices of the row. >> >> 7 is because the star-stencil is used in 3D. >> >> I create my RHS vector using: >> >> *call >> VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >> * >> >> The values for the matrix and vector were calculated before PETSc logging >> so they don't come into play. >> >> They are also done in a similar fashion for matrix x and y. I still can't >> get it why solving the z momentum eqn takes so much time. Which portion >> should I focus on? >> >> Tks! >> >> >> Matt >> >> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 3/10/2012 5:59 PM, Jed Brown wrote: >>> >>> There is an inordinate amount of time being spent in VecScatterEnd(). >>> That sometimes indicates a very bad partition. Also, are your "48 cores" >>> real physical cores or just "logical cores" (look like cores to the >>> operating system, usually advertised as "threads" by the vendor, nothing >>> like cores in reality)? That can cause a huge load imbalance and very >>> confusing results as over-subscribed threads compete for shared resources. >>> Step it back to 24 threads and 12 threads, send log_summary for each. >>> >>> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: >>> >>>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>>> >>>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >>>> >>>>> Hi, >>>>> >>>>> I have combined the momentum linear eqns involving x,y,z into 1 large >>>>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>>>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>>>> procs. The log_summary is given below. I have some questions: >>>>> >>>>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>>>> does it says there are 4 matrix, 12 vector etc? >>>>> >>>> >>>> They are part of preconditioning. Are you sure you're using Hypre for >>>> this? It looks like you are using bjacobi/ilu. >>>> >>>> >>>>> >>>>> 2. I'm looking at the stages which take the longest time. It seems >>>>> that MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very >>>>> high ratios. The ratios of some others are also not too good (~ 1.6 - 2). >>>>> So are these stages the reason why my code is not scaling well? What can I >>>>> do to improve it? >>>>> >>>> >>>> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >>>> MatLUFactorNumeric, and VecNorm+VecDot. >>>> >>>> The high VecAssembly time might be due to generating a lot of entries >>>> off-process? >>>> >>>> In any case, this looks like an _extremely_ slow network, perhaps >>>> it's misconfigured? >>>> >>>> >>>> My cluster is configured with 48 procs per node. I re-run the case, >>>> using only 48 procs, thus there's no need to pass over a 'slow' >>>> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >>>> eqn respectively. I have also separated the x,y,z component of the momentum >>>> eqn to 3 separate linear eqns to debug the problem. >>>> >>>> Results show that stage "momentum_z" is taking a lot of time. I wonder >>>> if it has to do with the fact that I am partitioning my grids in the z >>>> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >>>> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >>>> >>>> I wonder why a lot of entries are generated off-process. >>>> >>>> I create my RHS vector using: >>>> >>>> *call >>>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>>> * >>>> >>>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>>> >>>> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >>>> >>>> I then insert the values into the vector using: >>>> >>>> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >>>> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >>>> ijk_xyz_end) , INSERT_VALUES , ierr)* >>>> >>>> What should I do to correct the problem? >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>>> >>>>> Btw, I insert matrix using: >>>>> >>>>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>>>> ** >>>>> ** II = ijk - 1** !Fortran shift to 0-based** >>>>> ** ** >>>>> ** call >>>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>>>> ** >>>>> ** >>>>> **end do* >>>>> >>>>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>>>> >>>>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>>>> >>>>> semi_mat_xyz has the corresponding values. >>>>> >>>>> and I insert vectors using: >>>>> >>>>> call >>>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>>> >>>>> Thanks! >>>>> >>>>> * >>>>> * >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>>> >>>>> You can measure the time spent in Hypre via PCApply and PCSetUp, but >>>>> you can't get finer grained integrated profiling because it was not set up >>>>> that way. >>>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>>>> >>>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>>> >>>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>>>> >>>>>>> 1. if I'm solving 3 linear equations using ksp, is the result given >>>>>>> in the log summary the total of the 3 linear eqns' performance? How can I >>>>>>> get the performance for each individual eqn? >>>>>>> >>>>>> >>>>>> Use logging stages: >>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>>> >>>>>> >>>>>>> 2. If I run my code for 10 time steps, does the log summary gives >>>>>>> the total or avg performance/ratio? >>>>>>> >>>>>> >>>>>> Total. >>>>>> >>>>>> >>>>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG >>>>>>> (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there any way I >>>>>>> can use PETSc's log summary to get HYPRE's performance? If I use boomerAMG >>>>>>> thru PETSc, can I get its performance? >>>>>> >>>>>> >>>>>> If you mean flops, only if you count them yourself and tell PETSc >>>>>> using >>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>>> >>>>>> This is the disadvantage of using packages that do not properly >>>>>> monitor things :) >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>>>> evaluating its performance, beside using PetscLogFlops? >>>>>> >>>>>> >>>>>>> -- >>>>>>> Yours sincerely, >>>>>>> >>>>>>> TAY wee-beng >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From porterma at lanl.gov Thu Oct 4 15:29:32 2012 From: porterma at lanl.gov (Mark Porter) Date: Thu, 04 Oct 2012 14:29:32 -0600 Subject: [petsc-users] petsc make test error Message-ID: <506DF1AC.5010306@lanl.gov> I am having trouble with the make test once petsc configure and I have run make all. The error is as follows: Running test examples to verify correct installation Using PETSC_DIR=/home/porterma/Applications/petsc-dev and PETSC_ARCH=linux-gnu-c-debug C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes --------------Error detected during compile or link!----------------------- See http://www.mcs.anl.gov/petsc/documentation/faq.html /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials ex5f --------------------------------------------------------------------------- /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -c -Wall -Wno-unused-variable -g -I/home/porterma/Applications/petsc-dev/include -I/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/include -o ex5f.o ex5f.F Warning: Nonconforming tab character in column 1 of line 3099 Warning: Nonconforming tab character in column 1 of line 5574 Warning: Nonconforming tab character in column 1 of line 7846 Warning: Nonconforming tab character in column 1 of line 10114 Warning: Nonconforming tab character in column 1 of line 12423 /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -Wall -Wno-unused-variable -g -o ex5f ex5f.o -L/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib -lpetsc -lpthread -Wl,-rpath,/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib -lflapack -lfblas -lhdf5_fortran -lhdf5 -lhdf5hl_fortran -lhdf5_hl -lz -L/usr/lib/gcc/x86_64-redhat-linux6E/4.4.6 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lrt -lnsl -lutil -lgcc_s -lpthread -ldl ex5f.o: In function `MAIN__': /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:94: undefined reference to `snescreate_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:112: undefined reference to `dmcreateglobalvector_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:113: undefined reference to `vecduplicate_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:124: undefined reference to `dmdagetcorners_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:126: undefined reference to `dmdagetghostcorners_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:145: undefined reference to `snessetdm_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:153: undefined reference to `snessetfromoptions_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:165: undefined reference to `snesgetiterationnumber_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:177: undefined reference to `vecdestroy_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:178: undefined reference to `vecdestroy_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:179: undefined reference to `snesdestroy_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:180: undefined reference to `dmdestroy_' ex5f.o: In function `forminitialguess_': /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:226: undefined reference to `dmgetlocalvector_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:239: undefined reference to `dmlocaltoglobalbegin_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:240: undefined reference to `dmlocaltoglobalend_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:241: undefined reference to `dmrestorelocalvector_' ex5f.o: In function `formjacobianlocal_': /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:462: undefined reference to `matsetvalueslocal_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:477: undefined reference to `matsetvalueslocal_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:481: undefined reference to `matassemblybegin_' /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:482: undefined reference to `matassemblyend_' collect2: ld returned 1 exit status make[3]: [ex5f] Error 1 (ignored) /bin/rm -f ex5f.o Completed test examples Any suggestions would be helpful. Thanks. porter -- Mark L. Porter, Ph.D. Earth& Environmental Sciences (EES-14) Los Alamos National Laboratory work: 505.606.1971 cell: 541.517.2761 porterma at lanl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Oct 4 15:34:22 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Oct 2012 16:34:22 -0400 Subject: [petsc-users] petsc make test error In-Reply-To: <506DF1AC.5010306@lanl.gov> References: <506DF1AC.5010306@lanl.gov> Message-ID: On Thu, Oct 4, 2012 at 4:29 PM, Mark Porter wrote: > I am having trouble with the make test once petsc configure and I have > run make all. The error is as follows: > 1) If you don't care about Fortran, everything is fine 2) If you have configured with Fortran data types, then the simple Fortran test does not work. I fixed this in the Python tests, but make is harder to deal with. Matt > Running test examples to verify correct installation > Using PETSC_DIR=/home/porterma/Applications/petsc-dev and > PETSC_ARCH=linux-gnu-c-debug > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI > process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI > processes > --------------Error detected during compile or link!----------------------- > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials ex5f > --------------------------------------------------------------------------- > /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -c > -Wall -Wno-unused-variable -g > -I/home/porterma/Applications/petsc-dev/include > -I/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/include -o > ex5f.o ex5f.F > Warning: Nonconforming tab character in column 1 of line 3099 > Warning: Nonconforming tab character in column 1 of line 5574 > Warning: Nonconforming tab character in column 1 of line 7846 > Warning: Nonconforming tab character in column 1 of line 10114 > Warning: Nonconforming tab character in column 1 of line 12423 > /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -Wall > -Wno-unused-variable -g -o ex5f ex5f.o > -L/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib -lpetsc > -lpthread > -Wl,-rpath,/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib > -lflapack -lfblas -lhdf5_fortran -lhdf5 -lhdf5hl_fortran -lhdf5_hl -lz > -L/usr/lib/gcc/x86_64-redhat-linux6E/4.4.6 -lmpi_f90 -lmpi_f77 -lgfortran > -lm -lm -lm -lm -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl > -lmpi -lrt -lnsl -lutil -lgcc_s -lpthread -ldl > ex5f.o: In function `MAIN__': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:94: > undefined reference to `snescreate_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:112: > undefined reference to `dmcreateglobalvector_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:113: > undefined reference to `vecduplicate_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:124: > undefined reference to `dmdagetcorners_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:126: > undefined reference to `dmdagetghostcorners_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:145: > undefined reference to `snessetdm_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:153: > undefined reference to `snessetfromoptions_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:165: > undefined reference to `snesgetiterationnumber_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:177: > undefined reference to `vecdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:178: > undefined reference to `vecdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:179: > undefined reference to `snesdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:180: > undefined reference to `dmdestroy_' > ex5f.o: In function `forminitialguess_': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:226: > undefined reference to `dmgetlocalvector_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:239: > undefined reference to `dmlocaltoglobalbegin_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:240: > undefined reference to `dmlocaltoglobalend_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:241: > undefined reference to `dmrestorelocalvector_' > ex5f.o: In function `formjacobianlocal_': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:462: > undefined reference to `matsetvalueslocal_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:477: > undefined reference to `matsetvalueslocal_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:481: > undefined reference to `matassemblybegin_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:482: > undefined reference to `matassemblyend_' > collect2: ld returned 1 exit status > make[3]: [ex5f] Error 1 (ignored) > /bin/rm -f ex5f.o > Completed test examples > > > Any suggestions would be helpful. > > Thanks. > porter > > > -- > Mark L. Porter, Ph.D. > Earth & Environmental Sciences (EES-14) > Los Alamos National Laboratory > work: 505.606.1971 > cell: 541.517.2761porterma at lanl.gov > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Thu Oct 4 16:02:47 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Thu, 4 Oct 2012 16:02:47 -0500 Subject: [petsc-users] petsc make test error In-Reply-To: <506DF1AC.5010306@lanl.gov> References: <506DF1AC.5010306@lanl.gov> Message-ID: <398730F7-5B40-458E-9665-DD5064A628D2@mcs.anl.gov> Send make.log to petsc-maint at mcs.anl.gov On Oct 4, 2012, at 3:29 PM, Mark Porter wrote: > I am having trouble with the make test once petsc configure and I have run make all. The error is as follows: > > Running test examples to verify correct installation > Using PETSC_DIR=/home/porterma/Applications/petsc-dev and PETSC_ARCH=linux-gnu-c-debug > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes > --------------Error detected during compile or link!----------------------- > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials ex5f > --------------------------------------------------------------------------- > /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -c -Wall -Wno-unused-variable -g -I/home/porterma/Applications/petsc-dev/include -I/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/include -o ex5f.o ex5f.F > Warning: Nonconforming tab character in column 1 of line 3099 > Warning: Nonconforming tab character in column 1 of line 5574 > Warning: Nonconforming tab character in column 1 of line 7846 > Warning: Nonconforming tab character in column 1 of line 10114 > Warning: Nonconforming tab character in column 1 of line 12423 > /home/porterma/Applications/petsc-dev/linux-gnu-c-debug/bin/mpif90 -Wall -Wno-unused-variable -g -o ex5f ex5f.o -L/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib -lpetsc -lpthread -Wl,-rpath,/home/porterma/Applications/petsc-dev/linux-gnu-c-debug/lib -lflapack -lfblas -lhdf5_fortran -lhdf5 -lhdf5hl_fortran -lhdf5_hl -lz -L/usr/lib/gcc/x86_64-redhat-linux6E/4.4.6 -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lrt -lnsl -lutil -lgcc_s -lpthread -ldl > ex5f.o: In function `MAIN__': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:94: undefined reference to `snescreate_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:112: undefined reference to `dmcreateglobalvector_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:113: undefined reference to `vecduplicate_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:124: undefined reference to `dmdagetcorners_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:126: undefined reference to `dmdagetghostcorners_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:145: undefined reference to `snessetdm_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:153: undefined reference to `snessetfromoptions_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:165: undefined reference to `snesgetiterationnumber_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:177: undefined reference to `vecdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:178: undefined reference to `vecdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:179: undefined reference to `snesdestroy_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:180: undefined reference to `dmdestroy_' > ex5f.o: In function `forminitialguess_': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:226: undefined reference to `dmgetlocalvector_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:239: undefined reference to `dmlocaltoglobalbegin_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:240: undefined reference to `dmlocaltoglobalend_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:241: undefined reference to `dmrestorelocalvector_' > ex5f.o: In function `formjacobianlocal_': > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:462: undefined reference to `matsetvalueslocal_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:477: undefined reference to `matsetvalueslocal_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:481: undefined reference to `matassemblybegin_' > /home/porterma/Applications/petsc-dev/src/snes/examples/tutorials/ex5f.F:482: undefined reference to `matassemblyend_' > collect2: ld returned 1 exit status > make[3]: [ex5f] Error 1 (ignored) > /bin/rm -f ex5f.o > Completed test examples > > > Any suggestions would be helpful. > > Thanks. > porter > > > > -- > Mark L. Porter, Ph.D. > Earth & Environmental Sciences (EES-14) > Los Alamos National Laboratory > work: 505.606.1971 > cell: 541.517.2761 > > porterma at lanl.gov From zonexo at gmail.com Thu Oct 4 16:38:23 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 04 Oct 2012 23:38:23 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> Message-ID: <506E01CF.8080109@gmail.com> On 4/10/2012 9:21 PM, Jed Brown wrote: > Can you send a picture of what your domain looks like and what shape > the part owned by a given processor looks like? Best would be to write > out the mesh with a variable marking the rank owning each vertex, then > do a color plot in Paraview or whatever you use to show the partition. > > VecScatterBegin/End is taking much more time than these, and really a > pretty unreasonable amount of time in general. Hi, I have attached my grid. I just use a simple paint software to color a particular partition. They are Cartesian grids. The center portion, where the wings are (in blue), have much closer spaced grids, due to the importance of the boundary layer. Hence although the partitions there seem thinner, the cells number for each partition is roughly the same. As mentioned earlier, the grid is partitioned in the Z direction. Hence, the variables are allocated as u(1:size_x,1:size_y,ksta:kend), where ksta,kend refer to the starting/ending indices in the z direction. Same for v,w etc. I hope it is clear enough now. > > On Thu, Oct 4, 2012 at 2:16 PM, Wee-Beng Tay > wrote: > > On 4/10/2012 5:11 PM, Matthew Knepley wrote: >> On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng > > wrote: >> >> On 4/10/2012 3:40 AM, Matthew Knepley wrote: >>> On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng >>> > wrote: >>> >>> Hi Jed, >>> >>> I believe they are real cores. Anyway, I have attached >>> the log summary for the 12/24/48 cores. I re-run a >>> smaller case because the large problem can't run with >>> 12cores. >>> >>> >>> Okay, look at VecScatterBegin/End for 24 and 48 cores (I am >>> guessing you have 4 16-core chips, but please figure this out). >>> The messages are logged in ScatterBegin, and the time is >>> logged in ScatterEnd. From 24 to 48 cores the time is cut in >>> half. >>> If you were only communicating the boundary, this is >>> completely backwards, so you are communicating a fair >>> fraction of ALL >>> the values in a subdomain. Figure out why your partition is >>> so screwed up and this will go away. >> >> What do you mean by "If you were only communicating the >> boundary, this is completely backwards, so you are >> communicating a fair fraction of ALL the values in a subdomain"? >> >> >> If you have 48 partitions instead of 24, you have a larger >> interface, so AssemblyEnd() should take >> slightly longer. However, your AssemblyEnd() takes HALF the time, >> which means its communicating >> much fewer values, which means you are not sending interface >> values, you are sending interior values, >> since the interior shrinks when you have more partitions. >> >> What this probably means is that your assembly routines are >> screwed up, and sending data all over the place. >> > Ok I got it now. Looking at the AssemblyEnd time, > > 12 procs > > MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 > 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 24 procs > > MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 > 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > 48 procs > > MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 > 6.0e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 > > > VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > VecAssemblyEnd time increases with procs, does it mean that there > is nothing wrong with it? > > On the other hand, MatAssemblyEnd time decreases with procs. So > that's where the problem lies, is that so? > > I'm still scanning my code but haven't found the error yet. It > seems strange because I inserted the matrix and vector exactly the > same way for x,y,z. The u,v,w are also allocated with the same > indices. Shouldn't the error be the same for x, y and z too? > > Trying to get some hints as to where else I need to look in my code... > > Tks > > > > > > > >> Matt >> >> I partition my domain in the z direction, as shown in the >> attached pic. The circled region is where the airfoils are. >> I'm using an immersed boundary method (IBM) code so the grid >> is all Cartesian. >> >> I created my Z matrix using: >> >> call >> MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) >> >> where ijk_sta / ijk_end are the starting/ending global >> indices of the row. >> >> 7 is because the star-stencil is used in 3D. >> >> I create my RHS vector using: >> >> /call >> VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ >> >> The values for the matrix and vector were calculated before >> PETSc logging so they don't come into play. >> >> They are also done in a similar fashion for matrix x and y. I >> still can't get it why solving the z momentum eqn takes so >> much time. Which portion should I focus on? >> >> Tks! >> >>> >>> Matt >>> >>> Yours sincerely, >>> >>> TAY wee-beng >>> >>> On 3/10/2012 5:59 PM, Jed Brown wrote: >>>> There is an inordinate amount of time being spent in >>>> VecScatterEnd(). That sometimes indicates a very bad >>>> partition. Also, are your "48 cores" real physical >>>> cores or just "logical cores" (look like cores to the >>>> operating system, usually advertised as "threads" by >>>> the vendor, nothing like cores in reality)? That can >>>> cause a huge load imbalance and very confusing results >>>> as over-subscribed threads compete for shared >>>> resources. Step it back to 24 threads and 12 threads, >>>> send log_summary for each. >>>> >>>> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng >>>> > wrote: >>>> >>>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>>>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng >>>>> > wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have combined the momentum linear eqns >>>>> involving x,y,z into 1 large matrix. The >>>>> Poisson eqn is solved using HYPRE strcut >>>>> format so it's not included. I run the code >>>>> for 50 timesteps (hence 50 kspsolve) using 96 >>>>> procs. The log_summary is given below. I have >>>>> some questions: >>>>> >>>>> 1. After combining the matrix, I should have >>>>> only 1 PETSc matrix. Why does it says there >>>>> are 4 matrix, 12 vector etc? >>>>> >>>>> >>>>> They are part of preconditioning. Are you sure >>>>> you're using Hypre for this? It looks like you are >>>>> using bjacobi/ilu. >>>>> >>>>> >>>>> 2. I'm looking at the stages which take the >>>>> longest time. It seems that MatAssemblyBegin, >>>>> VecNorm, VecAssemblyBegin, VecScatterEnd have >>>>> very high ratios. The ratios of some others >>>>> are also not too good (~ 1.6 - 2). So are >>>>> these stages the reason why my code is not >>>>> scaling well? What can I do to improve it? >>>>> >>>>> >>>>> 3/4 of the solve time is evenly balanced between >>>>> MatMult, MatSolve, MatLUFactorNumeric, and >>>>> VecNorm+VecDot. >>>>> >>>>> The high VecAssembly time might be due to >>>>> generating a lot of entries off-process? >>>>> >>>>> In any case, this looks like an _extremely_ slow >>>>> network, perhaps it's misconfigured? >>>> >>>> My cluster is configured with 48 procs per node. I >>>> re-run the case, using only 48 procs, thus there's >>>> no need to pass over a 'slow' interconnect. I'm now >>>> also using GAMG and BCGS for the poisson and >>>> momentum eqn respectively. I have also separated >>>> the x,y,z component of the momentum eqn to 3 >>>> separate linear eqns to debug the problem. >>>> >>>> Results show that stage "momentum_z" is taking a >>>> lot of time. I wonder if it has to do with the fact >>>> that I am partitioning my grids in the z direction. >>>> VecScatterEnd, MatMult are taking a lot of time. >>>> VecNormalize, VecScatterEnd, VecNorm, >>>> VecAssemblyBegin 's ratio are also not good. >>>> >>>> I wonder why a lot of entries are generated >>>> off-process. >>>> >>>> I create my RHS vector using: >>>> >>>> /call >>>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr)/ >>>> >>>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>>> >>>> /call >>>> MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)/ >>>> >>>> I then insert the values into the vector using: >>>> >>>> /call VecSetValues(b_rhs_semi_z , ijk_xyz_end - >>>> ijk_xyz_sta , (/ijk_xyz_sta : ijk_xyz_end - 1/) , >>>> q_semi_vect_z(ijk_xyz_sta + 1 : ijk_xyz_end) , >>>> INSERT_VALUES , ierr)/ >>>> >>>> What should I do to correct the problem? >>>> >>>> Thanks >>>> >>>> >>>>> >>>>> Btw, I insert matrix using: >>>>> >>>>> /do ijk=ijk_xyz_sta+1,ijk_xyz_end// >>>>> // >>>>> // II = ijk - 1//!Fortran shift to 0-based// >>>>> //// >>>>> //call >>>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)// >>>>> // >>>>> //end do/ >>>>> >>>>> where ijk_xyz_sta/ijk_xyz_end are the >>>>> starting/end index >>>>> >>>>> int_semi_xyz(ijk,1:7) stores the 7 column >>>>> global indices >>>>> >>>>> semi_mat_xyz has the corresponding values. >>>>> >>>>> and I insert vectors using: >>>>> >>>>> call >>>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>>> >>>>> Thanks! >>>>> >>>>> / >>>>> / >>>>> >>>>> Yours sincerely, >>>>> >>>>> TAY wee-beng >>>>> >>>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>>>> >>>>>> You can measure the time spent in Hypre via >>>>>> PCApply and PCSetUp, but you can't get finer >>>>>> grained integrated profiling because it was >>>>>> not set up that way. >>>>>> >>>>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" >>>>>> > >>>>>> wrote: >>>>>> >>>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY >>>>>>> wee-beng >>>>>> > wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm doing a log summary for my 3d >>>>>>> cfd code. I have some questions: >>>>>>> >>>>>>> 1. if I'm solving 3 linear equations >>>>>>> using ksp, is the result given in >>>>>>> the log summary the total of the 3 >>>>>>> linear eqns' performance? How can I >>>>>>> get the performance for each >>>>>>> individual eqn? >>>>>>> >>>>>>> >>>>>>> Use logging stages: >>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>>>> >>>>>>> 2. If I run my code for 10 time >>>>>>> steps, does the log summary gives >>>>>>> the total or avg performance/ratio? >>>>>>> >>>>>>> >>>>>>> Total. >>>>>>> >>>>>>> 3. Besides PETSc, I'm also using >>>>>>> HYPRE's native geometric MG (Struct) >>>>>>> to solve my Cartesian's grid CFD >>>>>>> poisson eqn. Is there any way I can >>>>>>> use PETSc's log summary to get >>>>>>> HYPRE's performance? If I use >>>>>>> boomerAMG thru PETSc, can I get its >>>>>>> performance? >>>>>>> >>>>>>> >>>>>>> If you mean flops, only if you count >>>>>>> them yourself and tell PETSc using >>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>>>> >>>>>>> This is the disadvantage of using >>>>>>> packages that do not properly monitor >>>>>>> things :) >>>>>>> >>>>>>> Matt >>>>>> So u mean if I use boomerAMG thru PETSc, >>>>>> there is no proper way of evaluating its >>>>>> performance, beside using PetscLogFlops? >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Yours sincerely, >>>>>>> >>>>>>> TAY wee-beng >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted >>>>>>> before they begin their experiments is >>>>>>> infinitely more interesting than any >>>>>>> results to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 3d_grid.jpg Type: image/jpeg Size: 160251 bytes Desc: not available URL: From knepley at gmail.com Thu Oct 4 16:40:40 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 4 Oct 2012 17:40:40 -0400 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506E01CF.8080109@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> Message-ID: On Thu, Oct 4, 2012 at 5:38 PM, TAY wee-beng wrote: > On 4/10/2012 9:21 PM, Jed Brown wrote: > > Can you send a picture of what your domain looks like and what shape the > part owned by a given processor looks like? Best would be to write out the > mesh with a variable marking the rank owning each vertex, then do a color > plot in Paraview or whatever you use to show the partition. > > VecScatterBegin/End is taking much more time than these, and really a > pretty unreasonable amount of time in general. > > > Hi, > > I have attached my grid. I just use a simple paint software to color a > particular partition. They are Cartesian grids. The center portion, where > the wings are (in blue), have much closer spaced grids, due to the > importance of the boundary layer. Hence although the partitions there seem > thinner, the cells number for each partition is roughly the same. > > As mentioned earlier, the grid is partitioned in the Z direction. Hence, > the variables are allocated as u(1:size_x,1:size_y,ksta:kend), where > ksta,kend refer to the starting/ending indices in the z direction. Same for > v,w etc. I hope it is clear enough now. > This is way too many emails on this list. As I said before, the Mom-Z solve is bad because the assembly of the operator is screwed up. You are communicating too many values. So, jsut go into your code and count how many off process entries you set. Matt > On Thu, Oct 4, 2012 at 2:16 PM, Wee-Beng Tay wrote: > >> On 4/10/2012 5:11 PM, Matthew Knepley wrote: >> >> On Thu, Oct 4, 2012 at 11:01 AM, TAY wee-beng wrote: >> >>> On 4/10/2012 3:40 AM, Matthew Knepley wrote: >>> >>> On Wed, Oct 3, 2012 at 4:05 PM, TAY wee-beng wrote: >>> >>>> Hi Jed, >>>> >>>> I believe they are real cores. Anyway, I have attached the log summary >>>> for the 12/24/48 cores. I re-run a smaller case because the large problem >>>> can't run with 12cores. >>>> >>> >>> Okay, look at VecScatterBegin/End for 24 and 48 cores (I am guessing >>> you have 4 16-core chips, but please figure this out). >>> The messages are logged in ScatterBegin, and the time is logged in >>> ScatterEnd. From 24 to 48 cores the time is cut in half. >>> If you were only communicating the boundary, this is completely >>> backwards, so you are communicating a fair fraction of ALL >>> the values in a subdomain. Figure out why your partition is so screwed >>> up and this will go away. >>> >>> >>> What do you mean by "If you were only communicating the boundary, this >>> is completely backwards, so you are communicating a fair fraction of ALL >>> the values in a subdomain"? >>> >> >> If you have 48 partitions instead of 24, you have a larger interface, >> so AssemblyEnd() should take >> slightly longer. However, your AssemblyEnd() takes HALF the time, which >> means its communicating >> much fewer values, which means you are not sending interface values, you >> are sending interior values, >> since the interior shrinks when you have more partitions. >> >> What this probably means is that your assembly routines are screwed up, >> and sending data all over the place. >> >> Ok I got it now. Looking at the AssemblyEnd time, >> >> 12 procs >> >> MatAssemblyEnd 145 1.0 1.6342e+01 1.8 0.00e+00 0.0 4.4e+01 6.0e+04 >> 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> VecAssemblyEnd 388 1.0 1.4472e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> 24 procs >> >> MatAssemblyEnd 145 1.0 1.1618e+01 2.4 0.00e+00 0.0 9.2e+01 6.0e+04 >> 8.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> VecAssemblyEnd 388 1.0 2.3527e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> 48 procs >> >> MatAssemblyEnd 145 1.0 7.4327e+00 2.4 0.00e+00 0.0 1.9e+02 6.0e+04 >> 8.0e+00 0 0 0 0 0 0 0 0 0 0 >> >> >> VecAssemblyEnd 388 1.0 2.8818e-03 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> VecAssemblyEnd time increases with procs, does it mean that there is >> nothing wrong with it? >> >> On the other hand, MatAssemblyEnd time decreases with procs. So that's >> where the problem lies, is that so? >> >> I'm still scanning my code but haven't found the error yet. It seems >> strange because I inserted the matrix and vector exactly the same way for >> x,y,z. The u,v,w are also allocated with the same indices. Shouldn't the >> error be the same for x, y and z too? >> >> Trying to get some hints as to where else I need to look in my code... >> >> Tks >> >> >> >> >> >> >> >> Matt >> >> >>> I partition my domain in the z direction, as shown in the attached pic. >>> The circled region is where the airfoils are. I'm using an immersed >>> boundary method (IBM) code so the grid is all Cartesian. >>> >>> I created my Z matrix using: >>> >>> call >>> MatCreateAIJ(MPI_COMM_WORLD,ijk_end-ijk_sta,ijk_end-ijk_sta,PETSC_DECIDE,PETSC_DECIDE,7,PETSC_NULL_INTEGER,7,PETSC_NULL_INTEGER,A_semi_z,ierr) >>> >>> where ijk_sta / ijk_end are the starting/ending global indices of the >>> row. >>> >>> 7 is because the star-stencil is used in 3D. >>> >>> I create my RHS vector using: >>> >>> *call >>> VecCreateMPI(MPI_COMM_WORLD,ijk_end-ijk_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>> * >>> >>> The values for the matrix and vector were calculated before PETSc >>> logging so they don't come into play. >>> >>> They are also done in a similar fashion for matrix x and y. I still >>> can't get it why solving the z momentum eqn takes so much time. Which >>> portion should I focus on? >>> >>> Tks! >>> >>> >>> Matt >>> >>> >>>> Yours sincerely, >>>> >>>> TAY wee-beng >>>> >>>> On 3/10/2012 5:59 PM, Jed Brown wrote: >>>> >>>> There is an inordinate amount of time being spent in VecScatterEnd(). >>>> That sometimes indicates a very bad partition. Also, are your "48 cores" >>>> real physical cores or just "logical cores" (look like cores to the >>>> operating system, usually advertised as "threads" by the vendor, nothing >>>> like cores in reality)? That can cause a huge load imbalance and very >>>> confusing results as over-subscribed threads compete for shared resources. >>>> Step it back to 24 threads and 12 threads, send log_summary for each. >>>> >>>> On Wed, Oct 3, 2012 at 8:08 AM, TAY wee-beng wrote: >>>> >>>>> On 2/10/2012 2:43 PM, Jed Brown wrote: >>>>> >>>>> On Tue, Oct 2, 2012 at 8:35 AM, TAY wee-beng wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have combined the momentum linear eqns involving x,y,z into 1 large >>>>>> matrix. The Poisson eqn is solved using HYPRE strcut format so it's not >>>>>> included. I run the code for 50 timesteps (hence 50 kspsolve) using 96 >>>>>> procs. The log_summary is given below. I have some questions: >>>>>> >>>>>> 1. After combining the matrix, I should have only 1 PETSc matrix. Why >>>>>> does it says there are 4 matrix, 12 vector etc? >>>>>> >>>>> >>>>> They are part of preconditioning. Are you sure you're using Hypre >>>>> for this? It looks like you are using bjacobi/ilu. >>>>> >>>>> >>>>>> >>>>>> 2. I'm looking at the stages which take the longest time. It seems >>>>>> that MatAssemblyBegin, VecNorm, VecAssemblyBegin, VecScatterEnd have very >>>>>> high ratios. The ratios of some others are also not too good (~ 1.6 - 2). >>>>>> So are these stages the reason why my code is not scaling well? What can I >>>>>> do to improve it? >>>>>> >>>>> >>>>> 3/4 of the solve time is evenly balanced between MatMult, MatSolve, >>>>> MatLUFactorNumeric, and VecNorm+VecDot. >>>>> >>>>> The high VecAssembly time might be due to generating a lot of >>>>> entries off-process? >>>>> >>>>> In any case, this looks like an _extremely_ slow network, perhaps >>>>> it's misconfigured? >>>>> >>>>> >>>>> My cluster is configured with 48 procs per node. I re-run the case, >>>>> using only 48 procs, thus there's no need to pass over a 'slow' >>>>> interconnect. I'm now also using GAMG and BCGS for the poisson and momentum >>>>> eqn respectively. I have also separated the x,y,z component of the momentum >>>>> eqn to 3 separate linear eqns to debug the problem. >>>>> >>>>> Results show that stage "momentum_z" is taking a lot of time. I wonder >>>>> if it has to do with the fact that I am partitioning my grids in the z >>>>> direction. VecScatterEnd, MatMult are taking a lot of time. VecNormalize, >>>>> VecScatterEnd, VecNorm, VecAssemblyBegin 's ratio are also not good. >>>>> >>>>> I wonder why a lot of entries are generated off-process. >>>>> >>>>> I create my RHS vector using: >>>>> >>>>> *call >>>>> VecCreateMPI(MPI_COMM_WORLD,ijk_xyz_end-ijk_xyz_sta,PETSC_DECIDE,b_rhs_semi_z,ierr) >>>>> * >>>>> >>>>> where ijk_xyz_sta and ijk_xyz_end are obtained from >>>>> >>>>> *call MatGetOwnershipRange(A_semi_z,ijk_xyz_sta,ijk_xyz_end,ierr)* >>>>> >>>>> I then insert the values into the vector using: >>>>> >>>>> *call VecSetValues(b_rhs_semi_z , ijk_xyz_end - ijk_xyz_sta , >>>>> (/ijk_xyz_sta : ijk_xyz_end - 1/) , q_semi_vect_z(ijk_xyz_sta + 1 : >>>>> ijk_xyz_end) , INSERT_VALUES , ierr)* >>>>> >>>>> What should I do to correct the problem? >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Btw, I insert matrix using: >>>>>> >>>>>> *do ijk=ijk_xyz_sta+1,ijk_xyz_end** >>>>>> ** >>>>>> ** II = ijk - 1** !Fortran shift to 0-based** >>>>>> ** ** >>>>>> ** call >>>>>> MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr) >>>>>> ** >>>>>> ** >>>>>> **end do* >>>>>> >>>>>> where ijk_xyz_sta/ijk_xyz_end are the starting/end index >>>>>> >>>>>> int_semi_xyz(ijk,1:7) stores the 7 column global indices >>>>>> >>>>>> semi_mat_xyz has the corresponding values. >>>>>> >>>>>> and I insert vectors using: >>>>>> >>>>>> call >>>>>> VecSetValues(b_rhs_semi_xyz,ijk_xyz_end_mz-ijk_xyz_sta_mz,(/ijk_xyz_sta_mz:ijk_xyz_end_mz-1/),q_semi_vect_xyz(ijk_xyz_sta_mz+1:ijk_xyz_end_mz),INSERT_VALUES,ierr) >>>>>> >>>>>> Thanks! >>>>>> >>>>>> * >>>>>> * >>>>>> >>>>>> Yours sincerely, >>>>>> >>>>>> TAY wee-beng >>>>>> >>>>>> On 30/9/2012 11:30 PM, Jed Brown wrote: >>>>>> >>>>>> You can measure the time spent in Hypre via PCApply and PCSetUp, but >>>>>> you can't get finer grained integrated profiling because it was not set up >>>>>> that way. >>>>>> On Sep 30, 2012 3:26 PM, "TAY wee-beng" wrote: >>>>>> >>>>>>> On 27/9/2012 1:44 PM, Matthew Knepley wrote: >>>>>>> >>>>>>> On Thu, Sep 27, 2012 at 3:49 AM, TAY wee-beng wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm doing a log summary for my 3d cfd code. I have some questions: >>>>>>>> >>>>>>>> 1. if I'm solving 3 linear equations using ksp, is the result given >>>>>>>> in the log summary the total of the 3 linear eqns' performance? How can I >>>>>>>> get the performance for each individual eqn? >>>>>>>> >>>>>>> >>>>>>> Use logging stages: >>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogStagePush.html >>>>>>> >>>>>>> >>>>>>>> 2. If I run my code for 10 time steps, does the log summary gives >>>>>>>> the total or avg performance/ratio? >>>>>>>> >>>>>>> >>>>>>> Total. >>>>>>> >>>>>>> >>>>>>>> 3. Besides PETSc, I'm also using HYPRE's native geometric MG >>>>>>>> (Struct) to solve my Cartesian's grid CFD poisson eqn. Is there any way I >>>>>>>> can use PETSc's log summary to get HYPRE's performance? If I use boomerAMG >>>>>>>> thru PETSc, can I get its performance? >>>>>>> >>>>>>> >>>>>>> If you mean flops, only if you count them yourself and tell PETSc >>>>>>> using >>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogFlops.html >>>>>>> >>>>>>> This is the disadvantage of using packages that do not properly >>>>>>> monitor things :) >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> So u mean if I use boomerAMG thru PETSc, there is no proper way of >>>>>>> evaluating its performance, beside using PetscLogFlops? >>>>>>> >>>>>>> >>>>>>>> -- >>>>>>>> Yours sincerely, >>>>>>>> >>>>>>>> TAY wee-beng >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Oct 4 16:44:47 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 4 Oct 2012 16:44:47 -0500 (CDT) Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> Message-ID: On Thu, 4 Oct 2012, Matthew Knepley wrote: > the Mom-Z solve is bad because the assembly of the operator is > screwed up. You are communicating too many values. So, jsut go into > your code and count how many off process entries you set. running with '-info' should give debug info regarding this communication [stack size used for messages, and the number of mallocs needed etc..] Satish From zonexo at gmail.com Fri Oct 5 10:08:35 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Fri, 05 Oct 2012 17:08:35 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> Message-ID: <506EF7F3.3040709@gmail.com> On 4/10/2012 11:44 PM, Satish Balay wrote: > On Thu, 4 Oct 2012, Matthew Knepley wrote: > >> the Mom-Z solve is bad because the assembly of the operator is >> screwed up. You are communicating too many values. So, jsut go into >> your code and count how many off process entries you set. > running with '-info' should give debug info regarding this > communication [stack size used for messages, and the number of mallocs > needed etc..] > > Satish Ok, I found my problem. I was using: PetscLogStage stage and the time for the poisson eqn got combined into that of the momentum_z solve. Hence the confusion in results. I have attached the new results. The time is taken up mainly by GAMG. I am using: -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg What are the possible ways to improve the efficiency or reduce the time? If I use HYPRE's geometric multigrid, performance is somewhat better, such that momentum_x/y/z accounts for a total of 15% computing time. From the log_summary, for the momentum eqns, it seems that MatAssembly is the main problem, is that so? -------------- next part -------------- 0.250000000000000 0.400000000000000 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000 0.500000000000000 1.00000000000000 0.300000000000000 0 -400000 AB,AA,BB -3.99000009451993 4.20550010475563 4.07500009855721 size_x,size_y,size_z 75 404 629 Warning - length difference between element and cell max_element_length,min_element_length,min_delta 0.125984138460047 3.703647122769601E-004 1.000000000000000E-002 maximum ngh_surfaces and ngh_vertics are 72 26 minimum ngh_surfaces and ngh_vertics are 1 1 IIB_I_cell_no_uvw_total1 14652 15150 15000 61272 60750 62100 1 0.00300000 0.33018343 0.41442094 1.19457187 0.49100420E-02 -0.13216209E+00 0.19023281E+08 2 0.00214949 0.08042523 0.51210792 1.22010993 -0.13701147E-03 -0.10344701E+00 0.19018108E+08 3 0.00203615 0.02146905 0.54566355 1.23244869 0.10544591E-05 -0.90478005E-01 0.19017334E+08 4 0.00200855 0.01354072 0.56450141 1.24354136 -0.62836065E-05 0.20489605E-01 0.19017160E+08 5 0.00199787 0.01065016 0.58015529 1.25235552 -0.13971205E-03 0.11199829E+00 0.19017088E+08 escape_time reached, so abort fx_p,fy_p,fz_p,fx_v,fy_v,fz_v,cd_cl_cs_explicit1,cd_cl_cs_implicit1 -2.636323997944099E-007 -2.23618995423546 -0.161689384600393 2.513651961966575E-007 -3.449431535817790E-003 1.492238456281484E-002 -1.226720359775232E-008 -2.23963938577127 -0.146767000037579 7.158703928325935E-009 -2.42687397020185 -7.458519403454597E-002 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_shared_rel named n12-05 with 48 processors, by wtay Fri Oct 5 16:19:16 2012 Using Petsc Development HG revision: 9883b54053eca13dd473a4711adfd309d1436b6e HG Date: Sun Sep 30 22:42:36 2012 -0500 Max Max/Min Avg Total Time (sec): 1.840e+03 1.00338 1.837e+03 Objects: 5.680e+02 1.00000 5.680e+02 Flops: 1.735e+11 1.26417 1.414e+11 6.787e+12 Flops/sec: 9.426e+07 1.26189 7.696e+07 3.694e+09 MPI Messages: 1.682e+05 3.29645 1.099e+05 5.273e+06 MPI Message Lengths: 8.645e+09 2.00779 7.622e+04 4.019e+11 MPI Reductions: 1.374e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 2.4790e+02 13.5% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 3.600e+01 0.3% 1: poisson: 1.4545e+03 79.2% 6.4730e+12 95.4% 5.245e+06 99.5% 7.496e+04 98.3% 1.146e+04 83.4% 2: momentum_x: 4.5177e+01 2.5% 1.0421e+11 1.5% 9.400e+03 0.2% 4.179e+02 0.5% 7.480e+02 5.4% 3: momentum_y: 4.4457e+01 2.4% 1.0494e+11 1.5% 9.400e+03 0.2% 4.236e+02 0.6% 7.480e+02 5.4% 4: momentum_z: 4.5353e+01 2.5% 1.0502e+11 1.5% 9.400e+03 0.2% 4.235e+02 0.6% 7.480e+02 5.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson MatMult 43722 1.0 1.0849e+03 1.3 1.14e+11 1.3 4.2e+06 8.9e+04 0.0e+00 52 62 80 93 0 66 65 80 95 0 3897 MatMultAdd 6728 1.0 8.7529e+01 2.9 8.16e+09 1.5 5.1e+05 1.8e+04 0.0e+00 3 4 10 2 0 3 4 10 2 0 3043 MatMultTranspose 6728 1.0 1.4643e+02 4.8 8.16e+09 1.5 5.1e+05 1.8e+04 0.0e+00 3 4 10 2 0 4 4 10 2 0 1819 MatSolve 3364 0.0 2.1961e-02 0.0 4.42e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 201 MatLUFactorSym 1 1.0 5.3883e-05 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 5.2929e-05 5.8 1.04e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 196 MatConvert 4 1.0 1.4656e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.7515e-01 2.3 1.58e+07 1.4 4.0e+02 8.3e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 3186 MatAssemblyBegin 69 1.0 5.6784e+00 1.9 0.00e+00 0.0 1.2e+03 1.4e+04 7.4e+01 0 0 0 0 1 0 0 0 0 1 0 MatAssemblyEnd 69 1.0 1.4912e+00 1.1 0.00e+00 0.0 6.7e+03 1.0e+04 2.0e+02 0 0 0 0 1 0 0 0 0 2 0 MatGetRow 961340 1.1 5.2880e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 0.0 1.4782e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 7.2956e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 6.3715e-01 1.2 0.00e+00 0.0 1.9e+04 1.8e+04 2.8e+02 0 0 0 0 2 0 0 0 0 2 0 MatAXPY 4 1.0 4.7550e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 1.1370e+00 1.0 1.10e+07 1.4 2.8e+03 3.7e+04 9.6e+01 0 0 0 0 1 0 0 0 0 1 352 MatMatMultSym 4 1.0 8.7142e-01 1.0 0.00e+00 0.0 2.4e+03 2.9e+04 8.8e+01 0 0 0 0 1 0 0 0 0 1 0 MatMatMultNum 4 1.0 2.6963e-01 1.1 1.10e+07 1.4 4.0e+02 8.3e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 1482 MatPtAP 4 1.0 3.5065e+00 1.0 4.77e+08 3.1 5.9e+03 6.9e+04 1.1e+02 0 0 0 0 1 0 0 0 0 1 2441 MatPtAPSymbolic 4 1.0 2.0355e+00 1.1 0.00e+00 0.0 5.3e+03 5.7e+04 1.0e+02 0 0 0 0 1 0 0 0 0 1 0 MatPtAPNumeric 4 1.0 1.4810e+00 1.0 4.77e+08 3.1 6.0e+02 1.7e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 5778 MatTrnMatMult 4 1.0 8.2675e+00 1.0 5.30e+08 4.6 2.7e+03 4.7e+05 1.2e+02 0 0 0 0 1 1 0 0 0 1 2706 MatGetLocalMat 20 1.0 4.2365e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 12 1.0 2.7735e-01 2.3 0.00e+00 0.0 2.8e+03 1.2e+05 1.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 4.6894e-02 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 1722 1.0 1.1443e+00 2.2 1.06e+08 1.1 0.0e+00 0.0e+00 1.7e+03 0 0 0 0 13 0 0 0 0 15 4026 KSPSetUp 11 1.0 1.4655e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 50 1.0 1.4493e+03 1.0 1.67e+11 1.3 5.2e+06 7.5e+04 1.1e+04 79 95 99 98 81 100100100100 97 4466 VecDot 1632 1.0 8.7316e+01 6.6 1.38e+09 1.1 0.0e+00 0.0e+00 1.6e+03 2 1 0 0 12 3 1 0 0 14 712 VecDotNorm2 816 1.0 8.7681e+01 5.3 2.77e+09 1.1 0.0e+00 0.0e+00 2.4e+03 3 2 0 0 18 3 2 0 0 21 1419 VecMDot 1722 1.0 1.0294e+00 3.8 5.30e+07 1.1 0.0e+00 0.0e+00 1.7e+03 0 0 0 0 13 0 0 0 0 15 2238 VecNorm 4274 1.0 4.4562e+01 7.8 7.45e+08 1.1 0.0e+00 0.0e+00 4.3e+03 1 0 0 0 31 1 1 0 0 37 751 VecScale 30320 1.0 2.6244e+01 3.3 3.24e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 5377 VecCopy 8514 1.0 9.8725e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 28819 1.0 9.3239e+00 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 55510 1.0 7.0225e+01 3.0 1.29e+10 1.1 0.0e+00 0.0e+00 0.0e+00 2 8 0 0 0 3 9 0 0 0 8026 VecAYPX 53824 1.0 9.9916e+01 2.7 8.08e+09 1.1 0.0e+00 0.0e+00 0.0e+00 4 5 0 0 0 5 5 0 0 0 3525 VecAXPBYCZ 1632 1.0 3.1461e+01 2.1 2.77e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 2 2 0 0 0 3955 VecWAXPY 1632 1.0 3.2483e+01 2.4 1.38e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 2 1 0 0 0 1915 VecMAXPY 3408 1.0 3.1750e-01 3.2 6.27e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 8575 VecAssemblyBegin 195 1.0 4.5760e+0012.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.8e+02 0 0 0 0 4 0 0 0 0 5 0 VecAssemblyEnd 195 1.0 1.6549e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 40412 1.0 1.3043e+02 3.6 4.86e+09 1.1 0.0e+00 0.0e+00 0.0e+00 4 3 0 0 0 4 3 0 0 0 1622 VecScatterBegin 57281 1.0 2.0484e+01 4.3 0.00e+00 0.0 5.2e+06 7.5e+04 0.0e+00 1 0 99 98 0 1 0100100 0 0 VecScatterEnd 57281 1.0 6.3320e+0214.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 22 0 0 0 0 0 VecSetRandom 4 1.0 4.1965e-02 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 3408 1.0 3.5365e+01116.9 1.61e+07 1.1 0.0e+00 0.0e+00 3.4e+03 0 0 0 0 25 0 0 0 0 30 20 PCSetUp 2 1.0 2.1552e+01 1.0 9.41e+08 1.3 4.2e+04 7.0e+04 1.2e+03 1 1 1 1 8 1 1 1 1 10 1933 PCSetUpOnBlocks 1682 1.0 2.8644e-03 2.1 1.04e+04 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 4 PCApply 1682 1.0 1.2035e+03 1.2 1.50e+11 1.3 5.0e+06 7.0e+04 5.1e+03 62 83 96 88 37 78 87 96 90 44 4672 PCGAMGgraph_AGG 4 1.0 2.6394e+00 1.0 1.10e+07 1.4 1.2e+03 4.2e+04 7.6e+01 0 0 0 0 1 0 0 0 0 1 151 PCGAMGcoarse_AGG 4 1.0 9.4073e+00 1.0 5.30e+08 4.6 2.4e+04 7.9e+04 4.8e+02 1 0 0 0 3 1 0 0 0 4 2378 PCGAMGProl_AGG 4 1.0 2.3864e+00 1.0 0.00e+00 0.0 3.2e+03 4.6e+04 1.1e+02 0 0 0 0 1 0 0 0 0 1 0 PCGAMGPOpt_AGG 4 1.0 3.6797e+00 1.0 2.53e+08 1.2 6.8e+03 6.4e+04 2.1e+02 0 0 0 0 2 0 0 0 0 2 2809 --- Event Stage 2: momentum_x MatMult 98 1.0 2.9512e+00 1.1 5.31e+08 1.2 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 6 23 98 99 0 8071 MatSolve 147 1.0 3.5473e+00 1.1 7.79e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 33 0 0 0 9840 MatLUFactorNum 49 1.0 4.3144e+00 1.2 4.32e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 19 0 0 0 4482 MatILUFactorSym 1 1.0 8.6374e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 49 1.0 6.6507e+0030.0 0.00e+00 0.0 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 7 0 0 0 13 0 MatAssemblyEnd 49 1.0 2.4696e+00 1.8 0.00e+00 0.0 1.9e+02 5.9e+04 8.0e+00 0 0 0 0 0 5 0 2 1 1 0 MatGetRowIJ 1 1.0 6.9141e-06 7.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 9.7661e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 98 1.0 3.3892e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 49 1.0 1.3279e+01 1.0 2.32e+09 1.1 9.2e+03 2.4e+05 3.5e+02 1 2 0 1 3 29100 98 99 47 7848 VecDot 98 1.0 5.0782e-01 2.5 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 7356 VecDotNorm2 49 1.0 5.2031e-01 2.1 1.66e+08 1.1 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 1 7 0 0 20 14359 VecNorm 98 1.0 7.8282e-01 6.4 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 4772 VecCopy 98 1.0 3.3748e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 295 1.0 6.4768e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 98 1.0 6.4953e-01 1.4 1.66e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 11502 VecWAXPY 98 1.0 5.8485e-01 1.2 8.31e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 6387 VecAssemblyBegin 98 1.0 1.3877e+00100.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 0 0 0 0 2 1 0 0 0 39 0 VecAssemblyEnd 98 1.0 1.2057e-03 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 98 1.0 6.0991e-02 2.7 0.00e+00 0.0 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 0 0 98 99 0 0 VecScatterEnd 98 1.0 3.4077e-01 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 98 1.0 4.4126e+00 1.2 4.32e+08 1.2 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 9 19 0 0 1 4382 PCSetUpOnBlocks 49 1.0 4.4121e+00 1.2 4.32e+08 1.2 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 9 19 0 0 0 4382 PCApply 147 1.0 3.7470e+00 1.1 7.79e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 33 0 0 0 9316 --- Event Stage 3: momentum_y MatMult 98 1.0 2.9856e+00 1.1 5.36e+08 1.2 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 6 23 98 99 0 8055 MatSolve 147 1.0 3.5550e+00 1.1 7.87e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 34 0 0 0 9913 MatLUFactorNum 49 1.0 4.6245e+00 1.2 4.36e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 19 0 0 0 4218 MatILUFactorSym 1 1.0 8.6295e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 49 1.0 4.5507e+00207.0 0.00e+00 0.0 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 5 0 0 0 13 0 MatAssemblyEnd 49 1.0 2.4393e+00 1.8 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00 0 0 0 0 0 5 0 2 1 1 0 MatGetRowIJ 1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 9.5730e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 98 1.0 3.3241e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 49 1.0 1.3589e+01 1.0 2.34e+09 1.1 9.2e+03 2.4e+05 3.5e+02 1 2 0 1 3 31100 98 99 47 7723 VecDot 98 1.0 5.2115e-01 2.5 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 7168 VecDotNorm2 49 1.0 5.1932e-01 2.1 1.66e+08 1.1 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 1 7 0 0 20 14386 VecNorm 98 1.0 8.4822e-01 6.8 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 4404 VecCopy 98 1.0 3.1552e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 295 1.0 6.1609e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 98 1.0 6.5669e-01 1.4 1.66e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 11377 VecWAXPY 98 1.0 5.8188e-01 1.2 8.31e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 6420 VecAssemblyBegin 98 1.0 1.3747e+0099.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 0 0 0 0 2 1 0 0 0 39 0 VecAssemblyEnd 98 1.0 1.2748e-03 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 98 1.0 6.1957e-02 3.0 0.00e+00 0.0 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 0 0 98 99 0 0 VecScatterEnd 98 1.0 3.4862e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 98 1.0 4.7222e+00 1.2 4.36e+08 1.2 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 10 19 0 0 1 4131 PCSetUpOnBlocks 49 1.0 4.7218e+00 1.2 4.36e+08 1.2 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 10 19 0 0 0 4131 PCApply 147 1.0 3.8046e+00 1.1 7.87e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 34 0 0 0 9262 --- Event Stage 4: momentum_z MatMult 98 1.0 3.0181e+00 1.2 5.37e+08 1.2 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 6 23 98 99 0 7974 MatSolve 147 1.0 3.5577e+00 1.1 7.87e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 34 0 0 0 9913 MatLUFactorNum 49 1.0 4.4279e+00 1.2 4.37e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 19 0 0 0 4413 MatILUFactorSym 1 1.0 8.8243e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 49 1.0 5.2133e+00110.2 0.00e+00 0.0 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 6 0 0 0 13 0 MatAssemblyEnd 49 1.0 2.4158e+00 1.8 0.00e+00 0.0 1.9e+02 6.0e+04 8.0e+00 0 0 0 0 0 5 0 2 1 1 0 MatGetRowIJ 1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 9.3498e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSetUp 98 1.0 3.3165e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 49 1.0 1.3420e+01 1.0 2.34e+09 1.1 9.2e+03 2.4e+05 3.5e+02 1 2 0 1 3 30100 98 99 47 7826 VecDot 98 1.0 5.2100e-01 2.5 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 7170 VecDotNorm2 49 1.0 5.3533e-01 2.2 1.66e+08 1.1 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 1 7 0 0 20 13956 VecNorm 98 1.0 7.6278e-01 5.2 8.31e+07 1.1 0.0e+00 0.0e+00 9.8e+01 0 0 0 0 1 1 4 0 0 13 4897 VecCopy 98 1.0 3.4060e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 295 1.0 6.4979e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPBYCZ 98 1.0 6.4095e-01 1.4 1.66e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 7 0 0 0 11656 VecWAXPY 98 1.0 5.7849e-01 1.1 8.31e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 4 0 0 0 6457 VecAssemblyBegin 98 1.0 1.3208e+0054.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.9e+02 0 0 0 0 2 1 0 0 0 39 0 VecAssemblyEnd 98 1.0 1.2825e-03 6.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 98 1.0 6.1303e-02 2.8 0.00e+00 0.0 9.2e+03 2.4e+05 0.0e+00 0 0 0 1 0 0 0 98 99 0 0 VecScatterEnd 98 1.0 3.7855e-01 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 98 1.0 4.5256e+00 1.2 4.37e+08 1.2 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 9 19 0 0 1 4317 PCSetUpOnBlocks 49 1.0 4.5250e+00 1.2 4.37e+08 1.2 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 9 19 0 0 0 4318 PCApply 147 1.0 3.8119e+00 1.1 7.87e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 8 34 0 0 0 9252 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 12 56 568064932 0 Krylov Solver 4 13 31728 0 Vector 8 85 129979032 0 Vector Scatter 0 12 12720 0 Index Set 0 16 5102648 0 Preconditioner 0 13 12540 0 Viewer 1 0 0 0 --- Event Stage 1: poisson Matrix 117 76 368468980 0 Matrix Coarsen 4 4 2480 0 Krylov Solver 10 4 120512 0 Vector 225 175 107847696 0 Vector Scatter 31 22 23320 0 Index Set 81 74 737472 0 Preconditioner 11 4 3456 0 PetscRandom 4 4 2464 0 --- Event Stage 2: momentum_x Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 120488 0 Preconditioner 2 0 0 0 --- Event Stage 3: momentum_y Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122104 0 Preconditioner 2 0 0 0 --- Event Stage 4: momentum_z Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1504 0 Vector Scatter 1 0 0 0 Index Set 5 2 122096 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 4.04358e-05 Average time for zero size MPI_Send(): 2.70406e-05 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Mon Oct 1 11:36:09 2012 Configure options: --with-mpi-dir=/opt/openmpi-1.5.3/ --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.3-dev_shared_rel --known-mpi-shared=1 --with-shared-libraries ----------------------------------------- Libraries compiled on Mon Oct 1 11:36:09 2012 on hpc12 Machine characteristics: Linux-2.6.32-279.1.1.el6.x86_64-x86_64-with-centos-6.3-Final Using PETSc directory: /home/wtay/Codes/petsc-dev Using PETSc arch: petsc-3.3-dev_shared_rel ----------------------------------------- Using C compiler: /opt/openmpi-1.5.3/bin/mpicc -fPIC -wd1572 -O3 ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /opt/openmpi-1.5.3/bin/mpif90 -fPIC -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/include -I/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/include -I/opt/openmpi-1.5.3/include ----------------------------------------- Using C linker: /opt/openmpi-1.5.3/bin/mpicc Using Fortran linker: /opt/openmpi-1.5.3/bin/mpif90 Using libraries: -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -L/home/wtay/Codes/petsc-dev/petsc-3.3-dev_shared_rel/lib -lHYPRE -Wl,-rpath,/opt/openmpi-1.5.3/lib -L/opt/openmpi-1.5.3/lib -Wl,-rpath,/opt/intelcpro-11.1.059/lib/intel64 -L/opt/intelcpro-11.1.059/lib/intel64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpi_cxx -lstdc++ -lpthread -Wl,-rpath,/opt/intelcpro-11.1.059/mkl/lib/em64t -L/opt/intelcpro-11.1.059/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lnsl -lutil -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -ldl ----------------------------------------- From jedbrown at mcs.anl.gov Fri Oct 5 10:17:49 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Oct 2012 10:17:49 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <506EF7F3.3040709@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> Message-ID: Now most of the time is spent in PCApply, though an inordinate amount is still in VecScatterEnd, indicating that either your partition is very bad or the machine has contention (either with other jobs, zombie processes, incorrect affinity, over-subscribed). In any case, you can try changing the smoother. For example, try skipping the Chebyshev iterations and just use one iteration of SOR... On Fri, Oct 5, 2012 at 10:08 AM, TAY wee-beng wrote: > On 4/10/2012 11:44 PM, Satish Balay wrote: > >> On Thu, 4 Oct 2012, Matthew Knepley wrote: >> >> the Mom-Z solve is bad because the assembly of the operator is >>> screwed up. You are communicating too many values. So, jsut go into >>> your code and count how many off process entries you set. >>> >> running with '-info' should give debug info regarding this >> communication [stack size used for messages, and the number of mallocs >> needed etc..] >> >> Satish >> > > Ok, I found my problem. I was using: > > PetscLogStage stage > > and the time for the poisson eqn got combined into that of the momentum_z > solve. Hence the confusion in results. > > I have attached the new results. The time is taken up mainly by GAMG. I am > using: > > -poisson_pc_gamg_agg_nsmooths 1 > -poisson_pc_type gamg > > What are the possible ways to improve the efficiency or reduce the time? > > If I use HYPRE's geometric multigrid, performance is somewhat better, such > that momentum_x/y/z accounts for a total of 15% computing time. > > From the log_summary, for the momentum eqns, it seems that MatAssembly is > the main problem, is that so? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Oct 5 11:51:15 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Fri, 05 Oct 2012 11:51:15 -0500 Subject: [petsc-users] A similar question of "Enquiry regarding log summary results" Message-ID: <506F1003.80105@gmail.com> Dear folks, I hope you're having a nice day. I've been reading the thread of "Enquiry regarding log summary results" and tried the similar things with -log_summary with my code. However, my output seems totally different from Dr. TAY Wee-beng's. I attached it here. My code adopts the /src/ksp/ksp/example/tutorial/ex45.c from Dr. Yang to solve the Poisson equation (twice). The "Event Stage 3" and "Event Stage 5" are just for "KSPSolve", for example: ierr = PetscLogStagePush(stages[2]); CHKERRQ(ierr); // Start Pressure Poisson Equation Solve ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr); ierr = PetscLogStagePop(); CHKERRQ(ierr); // Finish Pressure Poisson Equation Solve Therefore, solving the equation takes most of my times. In the procedure of solving the equation, VecMDot, MatMult, MatMultAdd, MatMultTranspose takes lots of time. ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 3: Pressure Solve KSPGMRESOrthog 8643 1.0 2.0476e+0313.1 5.84e+09 1.0 0.0e+00 0.0e+00 8.6e+03 17 13 0 0 17 39 29 0 0 36 270 KSPSetUp 5 1.0 1.0729e-05 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 4.0481e+03 1.0 2.05e+10 1.0 4.0e+07 1.2e+03 2.4e+04 44 46 47 44 46 100100100100100 477 VecMDot 8643 1.0 2.0424e+0313.5 2.92e+09 1.0 0.0e+00 0.0e+00 8.6e+03 17 7 0 0 17 39 14 0 0 36 135 VecNorm 14503 1.0 8.7184e+02 6.4 8.47e+08 1.0 0.0e+00 0.0e+00 1.5e+04 7 2 0 0 28 15 4 0 0 61 92 VecScale 14503 1.0 3.7836e+00 4.7 4.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 10598 VecCopy 5767 1.0 2.0257e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 34670 1.0 5.5035e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8741 1.0 2.9888e+00 4.5 5.10e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 3 0 0 0 16172 VecAYPX 2881 1.0 1.7167e+00 7.2 8.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4640 VecWAXPY 93 1.0 5.3297e-02 4.8 2.71e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4824 VecMAXPY 14503 1.0 1.5191e+01 3.1 3.42e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 17 0 0 0 21342 VecAssemblyBegin 5 1.0 3.7097e+0012.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 2.9087e-05 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 26022 1.0 1.1726e+01 4.2 0.00e+00 0.0 4.0e+07 1.2e+03 0.0e+00 0 0 47 44 0 0 0100100 0 0 VecScatterEnd 26022 1.0 3.5268e+03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 19 0 0 0 0 43 0 0 0 0 0 VecNormalize 11524 1.0 8.4214e+02 7.2 1.01e+09 1.0 0.0e+00 0.0e+00 1.2e+04 6 2 0 0 22 15 5 0 0 49 114 MatMult 14498 1.0 3.0545e+0382.7 5.50e+09 1.0 6.4e+06 7.6e+03 0.0e+00 9 12 8 44 0 21 27 16 98 0 169 MatMultAdd 2881 1.0 1.8177e+0218.6 1.09e+09 1.1 3.6e+06 1.5e+01 0.0e+00 1 2 4 0 0 2 5 9 0 0 560 MatMultTranspose 2881 1.0 7.1296e+0274.9 1.09e+09 1.1 3.6e+06 1.5e+01 0.0e+00 5 2 4 0 0 11 5 9 0 0 143 MatSolve 14405 1.0 3.9663e+01 3.3 4.56e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 1 22 0 0 0 10890 MatLUFactorNum 5 1.0 3.9389e-02 1.6 3.05e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 7322 MatILUFactorSym 5 1.0 4.2221e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 5 1.0 9.7752e-06 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 5 1.0 2.9373e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.3110e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 3470 PCSetUpOnBlocks 5762 1.0 9.9790e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 2890 PCApply 2881 1.0 3.3151e+03 1.0 1.37e+10 1.0 3.8e+07 1.0e+03 1.7e+04 36 31 46 36 33 82 67 97 80 73 390 BTW, how to really calculate the time that a function costs? For example, it shows that "VecMDot, 2.0424e+0313.5". Would that be 2.0424*10^313.5 sec? thanks, Alan -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex45 on a arch-linux2-c-debug named compute-7-12.local with 96 processors, by zlwei Thu Oct 4 18:16:59 2012 Using Petsc Development HG revision: 98bf11863c3be31b7c2af504314a500bc64d88c9 HG Date: Wed Aug 29 13:51:08 2012 -0500 Max Max/Min Avg Total Time (sec): 9.287e+03 1.00000 9.287e+03 Objects: 2.271e+03 1.04127 2.192e+03 Flops: 4.641e+10 1.08786 4.392e+10 4.216e+12 Flops/sec: 4.998e+06 1.08786 4.729e+06 4.540e+08 MPI Messages: 1.039e+06 1.42613 8.729e+05 8.380e+07 MPI Message Lengths: 1.534e+09 2.02516 1.321e+03 1.107e+11 MPI Reductions: 5.191e+04 1.00174 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 6.3501e+02 6.8% 4.6734e+08 0.0% 2.344e+04 0.0% 7.888e+01 6.0% 3.200e+01 0.1% 1: DMMG Setup: 3.6959e+00 0.0% 0.0000e+00 0.0% 3.768e+03 0.0% 4.090e-02 0.0% 3.500e+01 0.1% 2: Pressure RHS Setup: 7.5716e+01 0.8% 9.2365e+10 2.2% 3.556e+05 0.4% 7.402e+00 0.6% 1.106e+03 2.1% 3: Pressure Solve: 4.0469e+03 43.6% 1.9304e+12 45.8% 3.978e+07 47.5% 5.878e+02 44.5% 2.376e+04 45.8% 4: Corrector RHS Setup: 1.2268e+02 1.3% 9.2365e+10 2.2% 3.556e+05 0.4% 7.402e+00 0.6% 1.106e+03 2.1% 5: Corrector Solve: 4.4028e+03 47.4% 2.1006e+12 49.8% 4.328e+07 51.7% 6.396e+02 48.4% 2.580e+04 49.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecNorm 10 1.0 4.2983e+0037.8 5.84e+05 1.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 12 0 0 31 13 VecSet 1 1.0 2.5029e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 10 1.0 2.1505e-03 3.0 5.84e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 12 0 0 0 25713 VecScatterBegin 11 1.0 6.8583e-03 6.0 0.00e+00 0.0 5.8e+03 6.9e+03 0.0e+00 0 0 0 0 0 0 0 25 1 0 0 VecScatterEnd 11 1.0 4.6764e+00693.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 10 1.0 3.7633e+00181.3 3.79e+06 1.0 4.4e+03 7.6e+03 0.0e+00 0 0 0 0 0 0 76 19 1 0 95 --- Event Stage 1: DMMG Setup --- Event Stage 2: Pressure RHS Setup KSPGMRESOrthog 835 2.0 2.1537e+00 8.0 1.19e+09 3.5 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 46 0 0 0 19894 KSPSetUp 25 1.0 5.1173e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 0 1 0 0 0 12 0 KSPSolve 30 1.0 5.3262e+00 4.9 1.82e+09 2.7 0.0e+00 0.0e+00 1.5e+02 0 2 0 0 0 4 85 0 0 13 14779 VecMDot 835 2.0 1.2693e+00 7.9 5.97e+08 3.5 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 23 0 0 0 16877 VecNorm 865 1.9 1.7481e-01 9.8 4.25e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 3 0 0 0 14441 VecScale 865 1.9 2.4498e-02 2.5 2.12e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 51524 VecCopy 30 1.0 1.1885e-02 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 250 1.2 9.3647e-02 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 30 1.0 9.2101e-03 4.0 1.58e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 15871 VecMAXPY 865 1.9 1.0598e+00 9.0 6.38e+08 3.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 26 0 0 0 22458 VecAssemblyBegin 5 1.0 6.5455e-01 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 1 0 0 0 1 0 VecAssemblyEnd 5 1.0 2.6226e-05 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 865 1.9 1.8566e-01 6.0 6.37e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 4 0 0 0 20396 MatMult 835 2.0 1.2689e+00 4.7 2.53e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 15 0 0 0 11213 MatSolve 865 1.9 1.5827e+00 3.7 2.63e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 16 0 0 0 9541 MatLUFactorSym 5 1.0 1.4618e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 1 0 MatLUFactorNum 10 1.0 8.0587e-02 1.9 2.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 27888 MatILUFactorSym 5 1.0 4.6887e-02 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 5 1.0 1.9262e-03 2.8 7.91e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 37944 MatAssemblyBegin 65 1.0 4.2835e+00 8.0 0.00e+00 0.0 6.4e+03 1.6e+04 5.0e+01 0 0 0 0 0 3 0 2 17 5 0 MatAssemblyEnd 65 1.0 1.2531e+01 1.1 0.00e+00 0.0 5.8e+04 1.5e+02 1.2e+02 0 0 0 0 0 16 0 16 1 11 0 MatGetRowIJ 10 1.0 1.2584e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSubMatrice 20 1.0 7.2942e+0027.7 0.00e+00 0.0 1.1e+04 9.1e+03 9.5e+01 0 0 0 0 0 3 0 3 16 9 0 MatGetOrdering 10 1.0 4.9312e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 3 0 MatMatMult 5 1.0 2.5914e-02 4.8 7.40e+05 2.4 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 2422 MatPtAP 5 1.0 2.2527e+01 1.0 1.85e+08 3.1 7.3e+04 3.3e+03 1.4e+02 0 0 0 0 0 30 12 21 38 12 510 MatPtAPSymbolic 5 1.0 2.1820e+01 1.0 0.00e+00 0.0 6.7e+04 3.5e+03 1.2e+02 0 0 0 0 0 29 0 19 37 11 0 MatPtAPNumeric 5 1.0 1.0236e+00 1.6 1.85e+08 3.1 6.2e+03 7.4e+02 1.0e+01 0 0 0 0 0 1 12 2 1 1 11233 MatGetRedundant 5 1.0 2.4219e+01 1.2 0.00e+00 0.0 1.4e+05 1.1e+03 2.0e+01 0 0 0 0 0 29 0 38 25 2 0 MatGetLocalMat 5 1.0 1.9222e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 MatGetBrAoCol 5 1.0 1.7008e+0119.2 0.00e+00 0.0 6.6e+03 3.5e+04 1.0e+01 0 0 0 0 0 12 0 2 37 1 0 MatGetSymTrans 10 1.0 9.4867e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 7.3916e+01 1.0 1.92e+09 2.5 3.3e+05 1.8e+03 7.1e+02 1 2 0 1 1 95100 93 96 64 1250 PCApply 865 1.9 1.5850e+00 3.7 2.63e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 16 0 0 0 9527 --- Event Stage 3: Pressure Solve KSPGMRESOrthog 8643 1.0 2.0476e+0313.1 5.84e+09 1.0 0.0e+00 0.0e+00 8.6e+03 17 13 0 0 17 39 29 0 0 36 270 KSPSetUp 5 1.0 1.0729e-05 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 4.0481e+03 1.0 2.05e+10 1.0 4.0e+07 1.2e+03 2.4e+04 44 46 47 44 46 100100100100100 477 VecMDot 8643 1.0 2.0424e+0313.5 2.92e+09 1.0 0.0e+00 0.0e+00 8.6e+03 17 7 0 0 17 39 14 0 0 36 135 VecNorm 14503 1.0 8.7184e+02 6.4 8.47e+08 1.0 0.0e+00 0.0e+00 1.5e+04 7 2 0 0 28 15 4 0 0 61 92 VecScale 14503 1.0 3.7836e+00 4.7 4.23e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 10598 VecCopy 5767 1.0 2.0257e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 34670 1.0 5.5035e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 8741 1.0 2.9888e+00 4.5 5.10e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 3 0 0 0 16172 VecAYPX 2881 1.0 1.7167e+00 7.2 8.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4640 VecWAXPY 93 1.0 5.3297e-02 4.8 2.71e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4824 VecMAXPY 14503 1.0 1.5191e+01 3.1 3.42e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 17 0 0 0 21342 VecAssemblyBegin 5 1.0 3.7097e+0012.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 2.9087e-05 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 26022 1.0 1.1726e+01 4.2 0.00e+00 0.0 4.0e+07 1.2e+03 0.0e+00 0 0 47 44 0 0 0100100 0 0 VecScatterEnd 26022 1.0 3.5268e+03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 19 0 0 0 0 43 0 0 0 0 0 VecNormalize 11524 1.0 8.4214e+02 7.2 1.01e+09 1.0 0.0e+00 0.0e+00 1.2e+04 6 2 0 0 22 15 5 0 0 49 114 MatMult 14498 1.0 3.0545e+0382.7 5.50e+09 1.0 6.4e+06 7.6e+03 0.0e+00 9 12 8 44 0 21 27 16 98 0 169 MatMultAdd 2881 1.0 1.8177e+0218.6 1.09e+09 1.1 3.6e+06 1.5e+01 0.0e+00 1 2 4 0 0 2 5 9 0 0 560 MatMultTranspose 2881 1.0 7.1296e+0274.9 1.09e+09 1.1 3.6e+06 1.5e+01 0.0e+00 5 2 4 0 0 11 5 9 0 0 143 MatSolve 14405 1.0 3.9663e+01 3.3 4.56e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 1 22 0 0 0 10890 MatLUFactorNum 5 1.0 3.9389e-02 1.6 3.05e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 7322 MatILUFactorSym 5 1.0 4.2221e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 5 1.0 9.7752e-06 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 5 1.0 2.9373e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.3110e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 3470 PCSetUpOnBlocks 5762 1.0 9.9790e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 2890 PCApply 2881 1.0 3.3151e+03 1.0 1.37e+10 1.0 3.8e+07 1.0e+03 1.7e+04 36 31 46 36 33 82 67 97 80 73 390 --- Event Stage 4: Corrector RHS Setup KSPGMRESOrthog 835 2.0 2.1493e+00 7.9 1.19e+09 3.5 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 46 0 0 0 19935 KSPSetUp 25 1.0 5.1442e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+02 0 0 0 0 0 0 0 0 0 12 0 KSPSolve 30 1.0 5.3268e+00 4.7 1.82e+09 2.7 0.0e+00 0.0e+00 1.5e+02 0 2 0 0 0 2 85 0 0 13 14777 VecMDot 835 2.0 1.2651e+00 7.8 5.97e+08 3.5 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 23 0 0 0 16933 VecNorm 865 1.9 1.4096e-01 7.7 4.25e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 3 0 0 0 17909 VecScale 865 1.9 2.4702e-02 2.5 2.12e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 51099 VecCopy 30 1.0 1.1083e-02 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 250 1.2 1.6737e-0110.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 30 1.0 9.1493e-03 3.9 1.58e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 15977 VecMAXPY 865 1.9 1.1516e+00 9.5 6.38e+08 3.3 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 26 0 0 0 20668 VecAssemblyBegin 5 1.0 6.9304e-01 9.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 1 0 VecAssemblyEnd 5 1.0 3.0994e-05 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 865 1.9 1.5194e-01 4.8 6.37e+07 1.8 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 4 0 0 0 24922 MatMult 835 2.0 1.2717e+00 4.5 2.53e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 15 0 0 0 11188 MatSolve 865 1.9 1.5930e+00 3.7 2.63e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 16 0 0 0 9478 MatLUFactorSym 5 1.0 1.3718e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 1 0 MatLUFactorNum 10 1.0 1.5112e-01 3.6 2.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 14871 MatILUFactorSym 5 1.0 4.7347e-02 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 5 1.0 2.2573e-03 3.6 7.91e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 32378 MatAssemblyBegin 65 1.0 4.3617e+00 8.9 0.00e+00 0.0 6.4e+03 1.6e+04 5.0e+01 0 0 0 0 0 2 0 2 17 5 0 MatAssemblyEnd 65 1.0 9.5838e+00 1.1 0.00e+00 0.0 5.8e+04 1.5e+02 1.2e+02 0 0 0 0 0 7 0 16 1 11 0 MatGetRowIJ 10 1.0 1.9398e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetSubMatrice 20 1.0 7.1810e+0026.4 0.00e+00 0.0 1.1e+04 9.1e+03 9.5e+01 0 0 0 0 0 2 0 3 16 9 0 MatGetOrdering 10 1.0 5.8639e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 3 0 MatMatMult 5 1.0 2.5045e-02 4.9 7.40e+05 2.4 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 2506 MatPtAP 5 1.0 1.9980e+01 1.0 1.85e+08 3.1 7.3e+04 3.3e+03 1.4e+02 0 0 0 0 0 16 12 21 38 12 575 MatPtAPSymbolic 5 1.0 1.9330e+01 1.0 0.00e+00 0.0 6.7e+04 3.5e+03 1.2e+02 0 0 0 0 0 16 0 19 37 11 0 MatPtAPNumeric 5 1.0 1.0708e+00 1.7 1.85e+08 3.1 6.2e+03 7.4e+02 1.0e+01 0 0 0 0 0 1 12 2 1 1 10737 MatGetRedundant 5 1.0 2.3013e+01 1.1 0.00e+00 0.0 1.4e+05 1.1e+03 2.0e+01 0 0 0 0 0 18 0 38 25 2 0 MatGetLocalMat 5 1.0 1.8890e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 1 0 MatGetBrAoCol 5 1.0 1.7516e+0129.6 0.00e+00 0.0 6.6e+03 3.5e+04 1.0e+01 0 0 0 0 0 7 0 2 37 1 0 MatGetSymTrans 10 1.0 9.4180e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 6.7590e+01 1.0 1.92e+09 2.5 3.3e+05 1.8e+03 7.1e+02 1 2 0 1 1 54100 93 96 64 1367 PCApply 865 1.9 1.5955e+00 3.7 2.63e+08 1.9 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 16 0 0 0 9464 --- Event Stage 5: Corrector Solve KSPGMRESOrthog 9405 1.0 2.2276e+0312.9 6.35e+09 1.0 0.0e+00 0.0e+00 9.4e+03 19 14 0 0 18 39 29 0 0 36 270 KSPSetUp 5 1.0 1.1921e-05 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 4.4038e+03 1.0 2.23e+10 1.0 4.3e+07 1.2e+03 2.6e+04 47 50 52 48 50 100100100100100 477 VecMDot 9405 1.0 2.2221e+0313.3 3.17e+09 1.0 0.0e+00 0.0e+00 9.4e+03 18 7 0 0 18 39 14 0 0 36 135 VecNorm 15782 1.0 9.4888e+02 6.4 9.21e+08 1.0 0.0e+00 0.0e+00 1.6e+04 7 2 0 0 30 15 4 0 0 61 92 VecScale 15782 1.0 4.1324e+00 4.7 4.61e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 10559 VecCopy 6275 1.0 2.1976e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 37727 1.0 5.9847e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 9512 1.0 3.2965e+00 4.6 5.55e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 3 0 0 0 15955 VecAYPX 3135 1.0 1.8733e+00 7.2 9.15e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4627 VecWAXPY 102 1.0 5.8617e-02 4.8 2.98e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4811 VecMAXPY 15782 1.0 1.6524e+01 3.1 3.72e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 17 0 0 0 21351 VecAssemblyBegin 5 1.0 2.8475e+00 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 4.1962e-05 8.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 28317 1.0 1.3162e+01 4.3 0.00e+00 0.0 4.3e+07 1.2e+03 0.0e+00 0 0 52 48 0 0 0100100 0 0 VecScatterEnd 28317 1.0 3.8321e+03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 43 0 0 0 0 0 VecNormalize 12540 1.0 9.1638e+02 7.2 1.10e+09 1.0 0.0e+00 0.0e+00 1.3e+04 7 2 0 0 24 15 5 0 0 49 114 MatMult 15777 1.0 3.3229e+0381.6 5.99e+09 1.0 6.9e+06 7.6e+03 0.0e+00 10 13 8 48 0 21 27 16 98 0 169 MatMultAdd 3135 1.0 1.9759e+0218.9 1.19e+09 1.1 3.9e+06 1.5e+01 0.0e+00 1 3 5 0 0 2 5 9 0 0 560 MatMultTranspose 3135 1.0 7.7572e+0277.0 1.19e+09 1.1 3.9e+06 1.5e+01 0.0e+00 5 3 5 0 0 11 5 9 0 0 143 MatSolve 15675 1.0 4.3195e+01 3.2 4.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 11 0 0 0 1 22 0 0 0 10881 MatLUFactorNum 5 1.0 3.9492e-02 1.7 3.05e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 7303 MatILUFactorSym 5 1.0 4.3832e-02 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 5 1.0 1.0014e-0510.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 5 1.0 1.3327e-02 9.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.3063e-02 1.9 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 3472 PCSetUpOnBlocks 6270 1.0 1.0566e-01 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 2730 PCApply 3135 1.0 3.6073e+03 1.0 1.49e+10 1.0 4.2e+07 1.0e+03 1.9e+04 39 33 50 39 36 82 67 97 80 73 390 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Container 0 20 10960 0 Krylov Solver 0 61 589472 0 Vector 11 1252 255019784 0 Vector Scatter 0 94 97384 0 Matrix 0 160 185546300120 0 Distributed Mesh 0 22 8606680 0 Bipartite Graph 0 44 30096 0 Index Set 0 90 2213200 0 IS L to G Mapping 0 33 5681412 0 Preconditioner 0 60 56800 0 Viewer 1 0 0 0 --- Event Stage 1: DMMG Setup Krylov Solver 1 0 0 0 Vector 5 4 5856 0 Vector Scatter 4 0 0 0 Distributed Mesh 2 0 0 0 Bipartite Graph 4 0 0 0 Index Set 10 10 264444 0 IS L to G Mapping 3 0 0 0 --- Event Stage 2: Pressure RHS Setup Container 10 0 0 0 Krylov Solver 30 0 0 0 Vector 385 55 2416680 0 Vector Scatter 45 0 0 0 Matrix 105 30 29966460 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 155 125 2631340 0 IS L to G Mapping 15 0 0 0 Preconditioner 30 0 0 0 --- Event Stage 3: Pressure Solve Vector 290 0 0 0 Matrix 5 0 0 0 Index Set 15 0 0 0 --- Event Stage 4: Corrector RHS Setup Container 10 0 0 0 Krylov Solver 30 0 0 0 Vector 385 55 2416680 0 Vector Scatter 45 0 0 0 Matrix 105 30 29966460 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 155 125 2631340 0 IS L to G Mapping 15 0 0 0 Preconditioner 30 0 0 0 --- Event Stage 5: Corrector Solve Vector 290 0 0 0 Matrix 5 0 0 0 Index Set 15 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 2.14577e-07 Average time for MPI_Barrier(): 0.000852394 Average time for zero size MPI_Send(): 0.00120548 #PETSc Option Table entries: -ksp_rtol 1.0e-7 -ksp_type fgmres -log_summary -mg_levels_ksp_max_it 1 -mg_levels_ksp_type gmres -mg_levels_pc_type bjacobi -pc_type exotic #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Tue Sep 18 16:22:39 2012 Configure options: --with-debugging=0 --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich ----------------------------------------- Libraries compiled on Tue Sep 18 16:22:39 2012 on firefox.bioinfo.ittc.ku.edu Machine characteristics: Linux-2.6.18-92.1.13.el5-x86_64-with-redhat-5.2-Final Using PETSc directory: /bio/work1/zlwei/PETSc/petsc-dev Using PETSc arch: arch-linux2-c-debug ----------------------------------------- Using C compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include ----------------------------------------- Using C linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc Using Fortran linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lgcc_s -ldl ----------------------------------------- From jedbrown at mcs.anl.gov Fri Oct 5 12:00:18 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Oct 2012 12:00:18 -0500 Subject: [petsc-users] A similar question of "Enquiry regarding log summary results" In-Reply-To: <506F1003.80105@gmail.com> References: <506F1003.80105@gmail.com> Message-ID: On Fri, Oct 5, 2012 at 11:51 AM, Zhenglun (Alan) Wei wrote: > Dear folks, > I hope you're having a nice day. > I've been reading the thread of "Enquiry regarding log summary > results" and tried the similar things with -log_summary with my code. > However, my output seems totally different from Dr. TAY Wee-beng's. I > attached it here. My code adopts the /src/ksp/ksp/example/tutorial/**ex45.c > from Dr. Yang to solve the Poisson equation (twice). The "Event Stage 3" > and "Event Stage 5" are just for "KSPSolve", for example: > > ierr = PetscLogStagePush(stages[2]); CHKERRQ(ierr); // Start Pressure > Poisson Equation Solve > ierr = KSPSolve(ksp,PETSC_NULL,PETSC_**NULL);CHKERRQ(ierr); > ierr = PetscLogStagePop(); CHKERRQ(ierr); // Finish Pressure Poisson > Equation Solve > > Therefore, solving the equation takes most of my times. In the > procedure of solving the equation, VecMDot, MatMult, MatMultAdd, > MatMultTranspose takes lots of time. > > ------------------------------**------------------------------** > ------------------------------**------------------------------ > Event Count Time (sec) Flops > --- Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess Avg len > Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > ------------------------------**------------------------------** > ------------------------------**------------------------------ > > --- Event Stage 3: Pressure Solve > > KSPGMRESOrthog 8643 1.0 2.0476e+0313.1 5.84e+09 1.0 0.0e+00 0.0e+00 > 8.6e+03 17 13 0 0 17 39 29 0 0 36 270 > KSPSetUp 5 1.0 1.0729e-05 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 5 1.0 4.0481e+03 1.0 2.05e+10 1.0 4.0e+07 1.2e+03 > 2.4e+04 44 46 47 44 46 100100100100100 477 > VecMDot 8643 1.0 2.0424e+0313.5 2.92e+09 1.0 0.0e+00 0.0e+00 > 8.6e+03 17 7 0 0 17 39 14 0 0 36 135 > VecNorm 14503 1.0 8.7184e+02 6.4 8.47e+08 1.0 0.0e+00 0.0e+00 > 1.5e+04 7 2 0 0 28 15 4 0 0 61 92 > VecScale 14503 1.0 3.7836e+00 4.7 4.23e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 1 0 0 0 0 2 0 0 0 10598 > VecCopy 5767 1.0 2.0257e+00 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 34670 1.0 5.5035e+00 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 8741 1.0 2.9888e+00 4.5 5.10e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 1 0 0 0 0 3 0 0 0 16172 > VecAYPX 2881 1.0 1.7167e+00 7.2 8.41e+07 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 4640 > VecWAXPY 93 1.0 5.3297e-02 4.8 2.71e+06 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 4824 > VecMAXPY 14503 1.0 1.5191e+01 3.1 3.42e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 8 0 0 0 0 17 0 0 0 21342 > VecAssemblyBegin 5 1.0 3.7097e+0012.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 5 1.0 2.9087e-05 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 26022 1.0 1.1726e+01 4.2 0.00e+00 0.0 4.0e+07 1.2e+03 > 0.0e+00 0 0 47 44 0 0 0100100 0 0 > VecScatterEnd 26022 1.0 3.5268e+03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 19 0 0 0 0 43 0 0 0 0 0 > VecNormalize 11524 1.0 8.4214e+02 7.2 1.01e+09 1.0 0.0e+00 0.0e+00 > 1.2e+04 6 2 0 0 22 15 5 0 0 49 114 > MatMult 14498 1.0 3.0545e+0382.7 5.50e+09 1.0 6.4e+06 7.6e+03 > 0.0e+00 9 12 8 44 0 21 27 16 98 0 169 > This load imbalance is totally unreasonable. Either you have an atrocious partition or there is contention on the machine (over-subscribed or other things running). The load balance for MatMult() should be close to 1.0 and the MFlop/s (last column) should be at least 10000 or so. > MatMultAdd 2881 1.0 1.8177e+0218.6 1.09e+09 1.1 3.6e+06 1.5e+01 > 0.0e+00 1 2 4 0 0 2 5 9 0 0 560 > MatMultTranspose 2881 1.0 7.1296e+0274.9 1.09e+09 1.1 3.6e+06 1.5e+01 > 0.0e+00 5 2 4 0 0 11 5 9 0 0 143 > MatSolve 14405 1.0 3.9663e+01 3.3 4.56e+09 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 10 0 0 0 1 22 0 0 0 10890 > MatLUFactorNum 5 1.0 3.9389e-02 1.6 3.05e+06 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 7322 > MatILUFactorSym 5 1.0 4.2221e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 > 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 5 1.0 9.7752e-06 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 5 1.0 2.9373e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 > 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 5 1.0 8.3110e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 > 1.5e+01 0 0 0 0 0 0 0 0 0 0 3470 > PCSetUpOnBlocks 5762 1.0 9.9790e-02 2.0 3.05e+06 1.0 0.0e+00 0.0e+00 > 1.5e+01 0 0 0 0 0 0 0 0 0 0 2890 > PCApply 2881 1.0 3.3151e+03 1.0 1.37e+10 1.0 3.8e+07 1.0e+03 > 1.7e+04 36 31 46 36 33 82 67 97 80 73 390 > > BTW, how to really calculate the time that a function costs? For > example, it shows that "VecMDot, 2.0424e+0313.5". Would that be > 2.0424*10^313.5 sec? > No, the imbalance is just spilling over. The amount of time is 2042 seconds. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Oct 5 12:18:08 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 5 Oct 2012 12:18:08 -0500 Subject: [petsc-users] Fwd: code using CVODE? References: Message-ID: <2C1EE7EF-B85C-4B00-B9BE-620418F0AEF5@mcs.anl.gov> Begin forwarded message: > From: "Woodward, Carol S." > Subject: code using CVODE? > Date: September 26, 2012 3:36:26 PM CDT > To: Barry Smith > > Hi Barry, > > I have a collaborator, Mayya Tokman ? CCed above, who has been developing an ODE time integration code based on exponential integrators. She implemented her new methods using the interfaces to SUNDIALS, and is now looking for candidate codes to try out her methods. Since you guys have an interface to CVODE, I thought I?d see if you have any users with a CVODE integration code who may be willing to work with a collaborator to test a new integrator. > > Let me know if you know of someone. > > Thanks, > > --Carol -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Oct 5 14:33:18 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Fri, 05 Oct 2012 14:33:18 -0500 Subject: [petsc-users] A similar question of "Enquiry regarding log summary results" In-Reply-To: References: <506F1003.80105@gmail.com> Message-ID: <506F35FE.9070709@gmail.com> Dear Dr. Brown, Thanks for your reply. I checked my grid partition by output the corner points of the local domain. There is no problem according to that. I ran the same case with smaller grid size with 8 cores and find that the code occupy a lot of memory and slow if I use the original script file which is /mpiexec -np 8 ./ex45 -pc_type exotic -ksp_type fgmres -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi -ksp_rtol 1.0e-7 -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out/ If I simply use /mpiexec -np 8 ./ex45 -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out/ Everything works fine. Therefore, I did more tests and found out that the "-pc_type exotic" has some problem. Now I changed it to "-pc_type gamg"; like, /mpiexec -np 8 ./ex45 -pc_type gamg -ksp_type fgmres -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi -ksp_rtol 1.0e-7 -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out// / it is super fast now, the log file is attached. However, I have another question that it always show that "unpreconditioned " although I already put pc_type over there. " 0 KSP unpreconditioned resid norm 1.800295456670e+01 true resid norm 1.800295456670e+01 ||r(i)||/||b|| 1.000000000000e+00" However, if I changed the 'ksp_type' from 'fgmres' to 'cg', it would give me: "0 KSP preconditioned resid norm 1.117293929796e+02 true resid norm 1.800293604638e+01 ||r(i)||/||b|| 1.000000000000e+00" The finial question: what are the choice for the /'/-mg_levels_ksp_type'. Are they as the same as 'ksp_type'? thank you so much, Alan On 10/5/2012 12:00 PM, Jed Brown wrote: > On Fri, Oct 5, 2012 at 11:51 AM, Zhenglun (Alan) Wei > > wrote: > > Dear folks, > I hope you're having a nice day. > I've been reading the thread of "Enquiry regarding log summary > results" and tried the similar things with -log_summary with my > code. However, my output seems totally different from Dr. TAY > Wee-beng's. I attached it here. My code adopts the > /src/ksp/ksp/example/tutorial/ex45.c from Dr. Yang to solve the > Poisson equation (twice). The "Event Stage 3" and "Event Stage 5" > are just for "KSPSolve", for example: > > ierr = PetscLogStagePush(stages[2]); CHKERRQ(ierr); // Start > Pressure Poisson Equation Solve > ierr = KSPSolve(ksp,PETSC_NULL,PETSC_NULL);CHKERRQ(ierr); > ierr = PetscLogStagePop(); CHKERRQ(ierr); // Finish Pressure > Poisson Equation Solve > > Therefore, solving the equation takes most of my times. In > the procedure of solving the equation, VecMDot, MatMult, > MatMultAdd, MatMultTranspose takes lots of time. > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flops --- > Global --- --- Stage --- Total > Max Ratio Max Ratio Max Ratio Mess > Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 3: Pressure Solve > > KSPGMRESOrthog 8643 1.0 2.0476e+0313.1 5.84e+09 1.0 0.0e+00 > 0.0e+00 8.6e+03 17 13 0 0 17 39 29 0 0 36 270 > KSPSetUp 5 1.0 1.0729e-05 3.5 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 5 1.0 4.0481e+03 1.0 2.05e+10 1.0 4.0e+07 > 1.2e+03 2.4e+04 44 46 47 44 46 100100100100100 477 > VecMDot 8643 1.0 2.0424e+0313.5 2.92e+09 1.0 0.0e+00 > 0.0e+00 8.6e+03 17 7 0 0 17 39 14 0 0 36 135 > VecNorm 14503 1.0 8.7184e+02 6.4 8.47e+08 1.0 0.0e+00 > 0.0e+00 1.5e+04 7 2 0 0 28 15 4 0 0 61 92 > VecScale 14503 1.0 3.7836e+00 4.7 4.23e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 10598 > VecCopy 5767 1.0 2.0257e+00 3.8 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 34670 1.0 5.5035e+00 4.7 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAXPY 8741 1.0 2.9888e+00 4.5 5.10e+08 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 1 0 0 0 0 3 0 0 0 16172 > VecAYPX 2881 1.0 1.7167e+00 7.2 8.41e+07 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4640 > VecWAXPY 93 1.0 5.3297e-02 4.8 2.71e+06 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4824 > VecMAXPY 14503 1.0 1.5191e+01 3.1 3.42e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 8 0 0 0 0 17 0 0 0 21342 > VecAssemblyBegin 5 1.0 3.7097e+0012.5 0.00e+00 0.0 0.0e+00 > 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyEnd 5 1.0 2.9087e-05 5.8 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 26022 1.0 1.1726e+01 4.2 0.00e+00 0.0 4.0e+07 > 1.2e+03 0.0e+00 0 0 47 44 0 0 0100100 0 0 > VecScatterEnd 26022 1.0 3.5268e+03 3.5 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 19 0 0 0 0 43 0 0 0 0 0 > VecNormalize 11524 1.0 8.4214e+02 7.2 1.01e+09 1.0 0.0e+00 > 0.0e+00 1.2e+04 6 2 0 0 22 15 5 0 0 49 114 > MatMult 14498 1.0 3.0545e+0382.7 5.50e+09 1.0 6.4e+06 > 7.6e+03 0.0e+00 9 12 8 44 0 21 27 16 98 0 169 > > > This load imbalance is totally unreasonable. Either you have an > atrocious partition or there is contention on the machine > (over-subscribed or other things running). The load balance for > MatMult() should be close to 1.0 and the MFlop/s (last column) should > be at least 10000 or so. > > MatMultAdd 2881 1.0 1.8177e+0218.6 1.09e+09 1.1 3.6e+06 > 1.5e+01 0.0e+00 1 2 4 0 0 2 5 9 0 0 560 > MatMultTranspose 2881 1.0 7.1296e+0274.9 1.09e+09 1.1 3.6e+06 > 1.5e+01 0.0e+00 5 2 4 0 0 11 5 9 0 0 143 > MatSolve 14405 1.0 3.9663e+01 3.3 4.56e+09 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 10 0 0 0 1 22 0 0 0 10890 > MatLUFactorNum 5 1.0 3.9389e-02 1.6 3.05e+06 1.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 7322 > MatILUFactorSym 5 1.0 4.2221e-02 3.2 0.00e+00 0.0 0.0e+00 > 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 5 1.0 9.7752e-06 5.1 0.00e+00 0.0 0.0e+00 > 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetOrdering 5 1.0 2.9373e-03 1.9 0.00e+00 0.0 0.0e+00 > 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 5 1.0 8.3110e-02 2.0 3.05e+06 1.0 0.0e+00 > 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 3470 > PCSetUpOnBlocks 5762 1.0 9.9790e-02 2.0 3.05e+06 1.0 0.0e+00 > 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 2890 > PCApply 2881 1.0 3.3151e+03 1.0 1.37e+10 1.0 3.8e+07 > 1.0e+03 1.7e+04 36 31 46 36 33 82 67 97 80 73 390 > > BTW, how to really calculate the time that a function > costs? For example, it shows that "VecMDot, 2.0424e+0313.5". Would > that be 2.0424*10^313.5 sec? > > > No, the imbalance is just spilling over. The amount of time is 2042 > seconds. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 node = 11 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 11 xs = 114 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 12 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 12 xs = 152 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 28 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 28 xs = 152 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 36 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 36 xs = 152 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 44 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 44 xs = 152 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 43 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 43 xs = 114 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 27 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 27 xs = 114 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 19 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 19 xs = 114 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 3 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 3 xs = 114 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 35 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 35 xs = 114 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 2 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 2 xs = 76 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 0 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 0 xs = 0 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 18 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 18 xs = 76 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 10 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 10 xs = 76 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 8 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 8 xs = 0 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 32 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 32 xs = 0 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 33 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 33 xs = 38 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 1 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 1 xs = 38 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 16 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 16 xs = 0 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 24 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 24 xs = 0 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 40 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 34 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 34 xs = 76 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 26 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 26 xs = 76 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 40 xs = 0 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 42 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 42 xs = 76 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 13 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 13 xs = 189 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 21 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 21 xs = 189 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 45 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 45 xs = 189 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 37 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 37 xs = 189 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 38 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 38 xs = 226 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 39 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 39 xs = 263 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 47 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 47 xs = 263 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 25 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 25 xs = 38 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 41 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 41 xs = 38 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 17 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 17 xs = 38 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 29 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 29 xs = 189 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 31 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 31 xs = 263 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 9 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 9 xs = 38 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 20 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 20 xs = 152 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 15 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 15 xs = 263 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 23 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 23 xs = 263 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 4 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 4 xs = 152 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 6 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 6 xs = 226 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 7 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 7 xs = 263 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 22 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 22 xs = 226 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 30 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 30 xs = 226 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 5 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 5 xs = 189 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 14 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 14 xs = 226 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 46 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 46 xs = 226 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 rank 18 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 35 front: 59 rank = 18, NbrRanks are: L: 17, R: 19, B:10, T:26, A:2, F:34!! rank = 18: xs = 76, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 18: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 18: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 10 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 35 front: 32 rank = 10, NbrRanks are: L: 9, R: 11, B:2, T:18, A:-6, F:26!! rank = 10: xs = 76, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 10: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 10: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 46 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank 9 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 35 front: 32 rank = 46, NbrRanks are: L: 45, R: 47, B:38, T:54, A:30, F:62!! rank = 9, NbrRanks are: L: 8, R: 10, B:1, T:17, A:-7, F:25!! rank = 46: xs = 226, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 9: xs = 38, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 46: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 9: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 46: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 9: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 11 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 11, NbrRanks are: L: 10, R: 12, B:3, T:19, A:-5, F:27!! rank = 11: xs = 114, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 11: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 11: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 45 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 45, NbrRanks are: L: 44, R: 46, B:37, T:53, A:29, F:61!! rank 6 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 6, NbrRanks are: L: 5, R: 7, B:-2, T:14, A:-10, F:22!! rank = 6: xs = 226, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 6: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 6: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 45: xs = 189, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 45: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 45: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 47 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank 8 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 35 front: 32 rank = 8, NbrRanks are: L: 7, R: 9, B:0, T:16, A:-8, F:24!! rank = 8: xs = 0, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 8: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 8: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 44 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 44, NbrRanks are: L: 43, R: 45, B:36, T:52, A:28, F:60!! rank = 44: xs = 152, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 44: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 44: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 47, NbrRanks are: L: 46, R: 48, B:39, T:55, A:31, F:63!! rank = 47: xs = 263, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 47: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 47: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 16 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 35 front: 59 rank = 16, NbrRanks are: L: 15, R: 17, B:8, T:24, A:0, F:32!! rank = 16: xs = 0, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 16: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 16: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 17 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 35 front: 59 rank = 17, NbrRanks are: L: 16, R: 18, B:9, T:25, A:1, F:33!! rank = 17: xs = 38, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 17: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 17: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 19 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 19, NbrRanks are: L: 18, R: 20, B:11, T:27, A:3, F:35!! rank = 19: xs = 114, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 19: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 19: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 4 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 4, NbrRanks are: L: 3, R: 5, B:-4, T:12, A:-12, F:20!! rank = 4: xs = 152, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 4: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 4: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 5 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 5, NbrRanks are: L: 4, R: 6, B:-3, T:13, A:-11, F:21!! rank = 5: xs = 189, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 5: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 5: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 7 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 7, NbrRanks are: L: 6, R: 8, B:-1, T:15, A:-9, F:23!! rank = 7: xs = 263, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 7: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 7: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 38 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 38, NbrRanks are: L: 37, R: 39, B:30, T:46, A:22, F:54!! rank = 38: xs = 226, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 38: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 38: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 14 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 14, NbrRanks are: L: 13, R: 15, B:6, T:22, A:-2, F:30!! rank = 14: xs = 226, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 14: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 14: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 22 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 22, NbrRanks are: L: 21, R: 23, B:14, T:30, A:6, F:38!! rank = 22: xs = 226, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 22: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 22: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 37 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 37, NbrRanks are: L: 36, R: 38, B:29, T:45, A:21, F:53!! rank = 37: xs = 189, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank 13 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank 21 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank 0 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 35 front: 32 rank = 0, NbrRanks are: L: -1, R: 1, B:-8, T:8, A:-16, F:16!! rank = 0: xs = 0, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 0: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 0: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank = 37: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 37: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 13, NbrRanks are: L: 12, R: 14, B:5, T:21, A:-3, F:29!! rank = 13: xs = 189, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 13: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 21, NbrRanks are: L: 20, R: 22, B:13, T:29, A:5, F:37!! rank = 21: xs = 189, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 21: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank 2 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 35 front: 32 rank = 2, NbrRanks are: L: 1, R: 3, B:-6, T:10, A:-14, F:18!! rank = 2: xs = 76, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 2: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 2: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank = 13: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 21: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 3 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 3, NbrRanks are: L: 2, R: 4, B:-5, T:11, A:-13, F:19!! rank = 3: xs = 114, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 3: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 3: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 20 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 20, NbrRanks are: L: 19, R: 21, B:12, T:28, A:4, F:36!! rank = 20: xs = 152, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 20: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 20: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 12 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 12, NbrRanks are: L: 11, R: 13, B:4, T:20, A:-4, F:28!! rank = 12: xs = 152, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank 15 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 15, NbrRanks are: L: 14, R: 16, B:7, T:23, A:-1, F:31!! rank = 15: xs = 263, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 12: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 12: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 15: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 15: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 24 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 35 front: 59 rank = 24, NbrRanks are: L: 23, R: 25, B:16, T:32, A:8, F:40!! rank = 24: xs = 0, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 24: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 24: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 36 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 36, NbrRanks are: L: 35, R: 37, B:28, T:44, A:20, F:52!! rank = 36: xs = 152, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 36: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 36: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 40 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 64 front: 59 rank = 40, NbrRanks are: L: 39, R: 41, B:32, T:48, A:24, F:56!! rank = 40: xs = 0, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 40: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 40: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 28 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 28, NbrRanks are: L: 27, R: 29, B:20, T:36, A:12, F:44!! rank = 28: xs = 152, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 28: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 28: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 23 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 23, NbrRanks are: L: 22, R: 24, B:15, T:31, A:7, F:39!! rank = 23: xs = 263, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 23: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 23: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 25 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 35 front: 59 rank = 25, NbrRanks are: L: 24, R: 26, B:17, T:33, A:9, F:41!! rank = 25: xs = 38, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 25: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 25: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 39 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 39, NbrRanks are: L: 38, R: 40, B:31, T:47, A:23, F:55!! rank = 39: xs = 263, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 39: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 39: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 41 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 64 front: 59 rank = 41, NbrRanks are: L: 40, R: 42, B:33, T:49, A:25, F:57!! rank = 41: xs = 38, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 41: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 41: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 29 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 29, NbrRanks are: L: 28, R: 30, B:21, T:37, A:13, F:45!! rank = 29: xs = 189, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 29: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 29: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 26 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 35 front: 59 rank = 26, NbrRanks are: L: 25, R: 27, B:18, T:34, A:10, F:42!! rank = 26: xs = 76, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 26: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 26: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 42 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 64 front: 59 rank = 42, NbrRanks are: L: 41, R: 43, B:34, T:50, A:26, F:58!! rank = 42: xs = 76, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 42: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 42: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 30 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 30, NbrRanks are: L: 29, R: 31, B:22, T:38, A:14, F:46!! rank = 30: xs = 226, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 30: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 30: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 27 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 27, NbrRanks are: L: 26, R: 28, B:19, T:35, A:11, F:43!! rank = 27: xs = 114, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 27: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 27: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 43 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 43, NbrRanks are: L: 42, R: 44, B:35, T:51, A:27, F:59!! rank = 43: xs = 114, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 43: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 43: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 31 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 31, NbrRanks are: L: 30, R: 32, B:23, T:39, A:15, F:47!! rank = 31: xs = 263, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 31: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 31: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 1 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 35 front: 32 rank = 1, NbrRanks are: L: 0, R: 2, B:-7, T:9, A:-15, F:17!! rank = 1: xs = 38, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 1: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 1: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 32 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 64 front: 59 rank = 32, NbrRanks are: L: 31, R: 33, B:24, T:40, A:16, F:48!! rank = 32: xs = 0, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 32: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 32: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 33 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 64 front: 59 rank = 33, NbrRanks are: L: 32, R: 34, B:25, T:41, A:17, F:49!! rank = 33: xs = 38, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 33: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 33: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 34 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 64 front: 59 rank = 34, NbrRanks are: L: 33, R: 35, B:26, T:42, A:18, F:50!! rank = 34: xs = 76, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 34: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 34: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 35 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 35, NbrRanks are: L: 34, R: 36, B:27, T:43, A:19, F:51!! rank = 35: xs = 114, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 35: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 35: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 0 KSP unpreconditioned resid norm 2.823135537733e+00 true resid norm 2.823135537733e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.154307416984e-01 true resid norm 2.154307416984e-01 ||r(i)||/||b|| 7.630903257002e-02 2 KSP unpreconditioned resid norm 6.616308524617e-02 true resid norm 6.616308524617e-02 ||r(i)||/||b|| 2.343602861494e-02 3 KSP unpreconditioned resid norm 3.244770285497e-02 true resid norm 3.244770285497e-02 ||r(i)||/||b|| 1.149349807024e-02 4 KSP unpreconditioned resid norm 2.037399364799e-02 true resid norm 2.037399364799e-02 ||r(i)||/||b|| 7.216796138789e-03 5 KSP unpreconditioned resid norm 1.115303682743e-02 true resid norm 1.115303682743e-02 ||r(i)||/||b|| 3.950584971341e-03 6 KSP unpreconditioned resid norm 7.349490089262e-03 true resid norm 7.349490089262e-03 ||r(i)||/||b|| 2.603307560346e-03 7 KSP unpreconditioned resid norm 5.032355046504e-03 true resid norm 5.032355046504e-03 ||r(i)||/||b|| 1.782541071530e-03 8 KSP unpreconditioned resid norm 3.412030504773e-03 true resid norm 3.412030504773e-03 ||r(i)||/||b|| 1.208596066030e-03 9 KSP unpreconditioned resid norm 2.506624879726e-03 true resid norm 2.506624879726e-03 ||r(i)||/||b|| 8.878868358331e-04 10 KSP unpreconditioned resid norm 1.911583249761e-03 true resid norm 1.911583249761e-03 ||r(i)||/||b|| 6.771135229646e-04 11 KSP unpreconditioned resid norm 1.450523741360e-03 true resid norm 1.450523741359e-03 ||r(i)||/||b|| 5.137988318210e-04 12 KSP unpreconditioned resid norm 1.165820441830e-03 true resid norm 1.165820441830e-03 ||r(i)||/||b|| 4.129523454500e-04 13 KSP unpreconditioned resid norm 9.075966183811e-04 true resid norm 9.075966183811e-04 ||r(i)||/||b|| 3.214853152640e-04 14 KSP unpreconditioned resid norm 7.190731823975e-04 true resid norm 7.190731823977e-04 ||r(i)||/||b|| 2.547072830145e-04 15 KSP unpreconditioned resid norm 5.738556352769e-04 true resid norm 5.738556352768e-04 ||r(i)||/||b|| 2.032688929054e-04 16 KSP unpreconditioned resid norm 4.542858026836e-04 true resid norm 4.542858026837e-04 ||r(i)||/||b|| 1.609153356656e-04 17 KSP unpreconditioned resid norm 3.691546556208e-04 true resid norm 3.691546556209e-04 ||r(i)||/||b|| 1.307605145722e-04 18 KSP unpreconditioned resid norm 3.023816670689e-04 true resid norm 3.023816670686e-04 ||r(i)||/||b|| 1.071084483997e-04 19 KSP unpreconditioned resid norm 2.446493662857e-04 true resid norm 2.446493662856e-04 ||r(i)||/||b|| 8.665873919820e-05 20 KSP unpreconditioned resid norm 2.063235929689e-04 true resid norm 2.063235929687e-04 ||r(i)||/||b|| 7.308313405824e-05 21 KSP unpreconditioned resid norm 1.690424749200e-04 true resid norm 1.690424749199e-04 ||r(i)||/||b|| 5.987756261099e-05 22 KSP unpreconditioned resid norm 1.401907850897e-04 true resid norm 1.401907850896e-04 ||r(i)||/||b|| 4.965783017354e-05 23 KSP unpreconditioned resid norm 1.144389156826e-04 true resid norm 1.144389156828e-04 ||r(i)||/||b|| 4.053610397137e-05 24 KSP unpreconditioned resid norm 8.920795008676e-05 true resid norm 8.920795008653e-05 ||r(i)||/||b|| 3.159889027438e-05 25 KSP unpreconditioned resid norm 6.909908488215e-05 true resid norm 6.909908488195e-05 ||r(i)||/||b|| 2.447600689319e-05 26 KSP unpreconditioned resid norm 5.266847004343e-05 true resid norm 5.266847004323e-05 ||r(i)||/||b|| 1.865601893330e-05 27 KSP unpreconditioned resid norm 4.064559966622e-05 true resid norm 4.064559966603e-05 ||r(i)||/||b|| 1.439732493278e-05 28 KSP unpreconditioned resid norm 3.171947373934e-05 true resid norm 3.171947373916e-05 ||r(i)||/||b|| 1.123554760840e-05 29 KSP unpreconditioned resid norm 2.462933869897e-05 true resid norm 2.462933869893e-05 ||r(i)||/||b|| 8.724107776529e-06 30 KSP unpreconditioned resid norm 1.822375473757e-05 true resid norm 1.822375473757e-05 ||r(i)||/||b|| 6.455146943529e-06 31 KSP unpreconditioned resid norm 1.574617811541e-05 true resid norm 1.574617811542e-05 ||r(i)||/||b|| 5.577549467591e-06 32 KSP unpreconditioned resid norm 1.307785144339e-05 true resid norm 1.307785144340e-05 ||r(i)||/||b|| 4.632385256962e-06 33 KSP unpreconditioned resid norm 1.142770280984e-05 true resid norm 1.142770280984e-05 ||r(i)||/||b|| 4.047876078603e-06 34 KSP unpreconditioned resid norm 9.171371046918e-06 true resid norm 9.171371046933e-06 ||r(i)||/||b|| 3.248647089150e-06 35 KSP unpreconditioned resid norm 7.576417541607e-06 true resid norm 7.576417541619e-06 ||r(i)||/||b|| 2.683688912684e-06 36 KSP unpreconditioned resid norm 6.341122445581e-06 true resid norm 6.341122445582e-06 ||r(i)||/||b|| 2.246127527648e-06 37 KSP unpreconditioned resid norm 5.520660583479e-06 true resid norm 5.520660583471e-06 ||r(i)||/||b|| 1.955506744074e-06 38 KSP unpreconditioned resid norm 4.758050788581e-06 true resid norm 4.758050788571e-06 ||r(i)||/||b|| 1.685378092896e-06 39 KSP unpreconditioned resid norm 4.119934417329e-06 true resid norm 4.119934417327e-06 ||r(i)||/||b|| 1.459347014078e-06 40 KSP unpreconditioned resid norm 3.423330512164e-06 true resid norm 3.423330512170e-06 ||r(i)||/||b|| 1.212598710340e-06 41 KSP unpreconditioned resid norm 2.843937477031e-06 true resid norm 2.843937477043e-06 ||r(i)||/||b|| 1.007368381373e-06 42 KSP unpreconditioned resid norm 2.247136771553e-06 true resid norm 2.247136771556e-06 ||r(i)||/||b|| 7.959719756711e-07 43 KSP unpreconditioned resid norm 1.767055534296e-06 true resid norm 1.767055534289e-06 ||r(i)||/||b|| 6.259194823171e-07 44 KSP unpreconditioned resid norm 1.330561337297e-06 true resid norm 1.330561337289e-06 ||r(i)||/||b|| 4.713062194518e-07 45 KSP unpreconditioned resid norm 1.002587660836e-06 true resid norm 1.002587660849e-06 ||r(i)||/||b|| 3.551326698448e-07 46 KSP unpreconditioned resid norm 7.571869187267e-07 true resid norm 7.571869187577e-07 ||r(i)||/||b|| 2.682077812551e-07 47 KSP unpreconditioned resid norm 5.557160212082e-07 true resid norm 5.557160212518e-07 ||r(i)||/||b|| 1.968435499551e-07 48 KSP unpreconditioned resid norm 4.235338619283e-07 true resid norm 4.235338619715e-07 ||r(i)||/||b|| 1.500225038121e-07 49 KSP unpreconditioned resid norm 3.213108718199e-07 true resid norm 3.213108718685e-07 ||r(i)||/||b|| 1.138134770981e-07 50 KSP unpreconditioned resid norm 2.465819253483e-07 true resid norm 2.465819253946e-07 ||r(i)||/||b|| 8.734328270778e-08 Linear solve converged due to CONVERGED_RTOL iterations 50 Residual norm 2.46582e-07 0 KSP unpreconditioned resid norm 2.464333785299e+00 true resid norm 2.464333785299e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.992196609304e-01 true resid norm 1.992196609304e-01 ||r(i)||/||b|| 8.084118398199e-02 2 KSP unpreconditioned resid norm 6.477029955211e-02 true resid norm 6.477029955211e-02 ||r(i)||/||b|| 2.628308711202e-02 3 KSP unpreconditioned resid norm 3.257461563671e-02 true resid norm 3.257461563671e-02 ||r(i)||/||b|| 1.321842675332e-02 4 KSP unpreconditioned resid norm 2.205871203061e-02 true resid norm 2.205871203061e-02 ||r(i)||/||b|| 8.951186792227e-03 5 KSP unpreconditioned resid norm 1.279229701970e-02 true resid norm 1.279229701970e-02 ||r(i)||/||b|| 5.190975790704e-03 6 KSP unpreconditioned resid norm 8.642476440830e-03 true resid norm 8.642476440830e-03 ||r(i)||/||b|| 3.507023477252e-03 7 KSP unpreconditioned resid norm 6.257653902253e-03 true resid norm 6.257653902254e-03 ||r(i)||/||b|| 2.539288281313e-03 8 KSP unpreconditioned resid norm 4.265808118282e-03 true resid norm 4.265808118282e-03 ||r(i)||/||b|| 1.731018802619e-03 9 KSP unpreconditioned resid norm 3.197638574989e-03 true resid norm 3.197638574989e-03 ||r(i)||/||b|| 1.297567153469e-03 10 KSP unpreconditioned resid norm 2.541200054047e-03 true resid norm 2.541200054046e-03 ||r(i)||/||b|| 1.031191500602e-03 11 KSP unpreconditioned resid norm 2.076630510557e-03 true resid norm 2.076630510557e-03 ||r(i)||/||b|| 8.426742038540e-04 12 KSP unpreconditioned resid norm 1.800573752044e-03 true resid norm 1.800573752044e-03 ||r(i)||/||b|| 7.306533566131e-04 13 KSP unpreconditioned resid norm 1.525854000374e-03 true resid norm 1.525854000374e-03 ||r(i)||/||b|| 6.191750522905e-04 14 KSP unpreconditioned resid norm 1.295503220832e-03 true resid norm 1.295503220832e-03 ||r(i)||/||b|| 5.257011970377e-04 15 KSP unpreconditioned resid norm 1.084042233066e-03 true resid norm 1.084042233065e-03 ||r(i)||/||b|| 4.398926150069e-04 16 KSP unpreconditioned resid norm 9.072748281531e-04 true resid norm 9.072748281522e-04 ||r(i)||/||b|| 3.681623137111e-04 17 KSP unpreconditioned resid norm 7.055698087217e-04 true resid norm 7.055698087231e-04 ||r(i)||/||b|| 2.863125981278e-04 18 KSP unpreconditioned resid norm 5.647143297836e-04 true resid norm 5.647143297835e-04 ||r(i)||/||b|| 2.291549680292e-04 19 KSP unpreconditioned resid norm 4.500753015708e-04 true resid norm 4.500753015704e-04 ||r(i)||/||b|| 1.826356901225e-04 20 KSP unpreconditioned resid norm 3.751165957785e-04 true resid norm 3.751165957780e-04 ||r(i)||/||b|| 1.522182579388e-04 21 KSP unpreconditioned resid norm 3.100897252663e-04 true resid norm 3.100897252655e-04 ||r(i)||/||b|| 1.258310571057e-04 22 KSP unpreconditioned resid norm 2.538816813673e-04 true resid norm 2.538816813667e-04 ||r(i)||/||b|| 1.030224407429e-04 23 KSP unpreconditioned resid norm 2.056390116417e-04 true resid norm 2.056390116429e-04 ||r(i)||/||b|| 8.344608708027e-05 24 KSP unpreconditioned resid norm 1.608371013495e-04 true resid norm 1.608371013504e-04 ||r(i)||/||b|| 6.526595638539e-05 25 KSP unpreconditioned resid norm 1.253589134032e-04 true resid norm 1.253589134030e-04 ||r(i)||/||b|| 5.086929138854e-05 26 KSP unpreconditioned resid norm 9.775671156099e-05 true resid norm 9.775671156060e-05 ||r(i)||/||b|| 3.966861637971e-05 27 KSP unpreconditioned resid norm 7.341573028295e-05 true resid norm 7.341573028263e-05 ||r(i)||/||b|| 2.979130940808e-05 28 KSP unpreconditioned resid norm 5.564380420345e-05 true resid norm 5.564380420310e-05 ||r(i)||/||b|| 2.257965399616e-05 29 KSP unpreconditioned resid norm 4.323591120259e-05 true resid norm 4.323591120220e-05 ||r(i)||/||b|| 1.754466519922e-05 30 KSP unpreconditioned resid norm 3.368099812126e-05 true resid norm 3.368099812126e-05 ||r(i)||/||b|| 1.366738480078e-05 31 KSP unpreconditioned resid norm 2.990017310739e-05 true resid norm 2.990017310738e-05 ||r(i)||/||b|| 1.213316689717e-05 32 KSP unpreconditioned resid norm 2.518721971644e-05 true resid norm 2.518721971645e-05 ||r(i)||/||b|| 1.022070137848e-05 33 KSP unpreconditioned resid norm 2.135321757357e-05 true resid norm 2.135321757358e-05 ||r(i)||/||b|| 8.664904771001e-06 34 KSP unpreconditioned resid norm 1.708173255141e-05 true resid norm 1.708173255150e-05 ||r(i)||/||b|| 6.931582342212e-06 35 KSP unpreconditioned resid norm 1.378692211645e-05 true resid norm 1.378692211657e-05 ||r(i)||/||b|| 5.594583898825e-06 36 KSP unpreconditioned resid norm 1.075085218635e-05 true resid norm 1.075085218651e-05 ||r(i)||/||b|| 4.362579554216e-06 37 KSP unpreconditioned resid norm 8.579543904874e-06 true resid norm 8.579543905050e-06 ||r(i)||/||b|| 3.481486134805e-06 38 KSP unpreconditioned resid norm 6.663024149842e-06 true resid norm 6.663024149966e-06 ||r(i)||/||b|| 2.703783144034e-06 39 KSP unpreconditioned resid norm 5.488872221286e-06 true resid norm 5.488872221412e-06 ||r(i)||/||b|| 2.227324989072e-06 40 KSP unpreconditioned resid norm 4.402005575508e-06 true resid norm 4.402005575670e-06 ||r(i)||/||b|| 1.786286257945e-06 41 KSP unpreconditioned resid norm 3.615398721527e-06 true resid norm 3.615398721731e-06 ||r(i)||/||b|| 1.467089703229e-06 42 KSP unpreconditioned resid norm 2.967489745085e-06 true resid norm 2.967489745333e-06 ||r(i)||/||b|| 1.204175247296e-06 43 KSP unpreconditioned resid norm 2.471957747140e-06 true resid norm 2.471957747409e-06 ||r(i)||/||b|| 1.003093721376e-06 44 KSP unpreconditioned resid norm 2.088449399093e-06 true resid norm 2.088449399350e-06 ||r(i)||/||b|| 8.474701811129e-07 45 KSP unpreconditioned resid norm 1.755508391147e-06 true resid norm 1.755508391304e-06 ||r(i)||/||b|| 7.123663205757e-07 46 KSP unpreconditioned resid norm 1.489882203160e-06 true resid norm 1.489882203306e-06 ||r(i)||/||b|| 6.045780860507e-07 47 KSP unpreconditioned resid norm 1.213783513857e-06 true resid norm 1.213783513883e-06 ||r(i)||/||b|| 4.925402237003e-07 48 KSP unpreconditioned resid norm 1.021510479234e-06 true resid norm 1.021510479145e-06 ||r(i)||/||b|| 4.145179055040e-07 49 KSP unpreconditioned resid norm 8.287946453344e-07 true resid norm 8.287946452356e-07 ||r(i)||/||b|| 3.363159041927e-07 50 KSP unpreconditioned resid norm 6.668156473414e-07 true resid norm 6.668156471973e-07 ||r(i)||/||b|| 2.705865784802e-07 51 KSP unpreconditioned resid norm 5.280582271016e-07 true resid norm 5.280582269962e-07 ||r(i)||/||b|| 2.142803179286e-07 52 KSP unpreconditioned resid norm 4.157123103826e-07 true resid norm 4.157123102940e-07 ||r(i)||/||b|| 1.686915598747e-07 53 KSP unpreconditioned resid norm 3.368064587348e-07 true resid norm 3.368064586986e-07 ||r(i)||/||b|| 1.366724186098e-07 54 KSP unpreconditioned resid norm 2.710364540227e-07 true resid norm 2.710364539492e-07 ||r(i)||/||b|| 1.099836619398e-07 55 KSP unpreconditioned resid norm 2.066135720810e-07 true resid norm 2.066135719884e-07 ||r(i)||/||b|| 8.384155312928e-08 Linear solve converged due to CONVERGED_RTOL iterations 55 Residual norm 2.06614e-07 Rank#24, time step = 0, continuity = 4.174955e-04 @ (41, 4, 4) Rank#27, time step = 0, continuity = 4.094066e+00 @ (4, 11, 17) Rank#10, time step = 0, continuity = 2.590009e-03 @ (39, 4, 35) Rank#20, time step = 0, continuity = 1.437608e-02 @ (4, 51, 20) Rank#11, time step = 0, continuity = 5.650684e-03 @ (4, 4, 35) Rank#21, time step = 0, continuity = 9.027686e-03 @ (4, 51, 20) Rank#1, time step = 0, continuity = 2.525042e-04 @ (4, 4, 4) Rank#16, time step = 0, continuity = 3.678162e-04 @ (41, 51, 4) Rank#4, time step = 0, continuity = 1.159767e-02 @ (4, 51, 35) Rank#8, time step = 0, continuity = 4.915367e-05 @ (41, 4, 35) Rank#34, time step = 0, continuity = 1.181571e-01 @ (40, 51, 4) Rank#0, time step = 0, continuity = 3.137227e-08 @ (41, 51, 4) Rank#17, time step = 0, continuity = 5.089529e-03 @ (41, 51, 4) Rank#5, time step = 0, continuity = 8.826453e-03 @ (4, 51, 35) Rank#2, time step = 0, continuity = 1.634919e-03 @ (4, 51, 35) Rank#6, time step = 0, continuity = 8.433646e-03 @ (4, 51, 35) Rank#12, time step = 0, continuity = 1.165931e-02 @ (4, 4, 35) Rank#13, time step = 0, continuity = 8.826511e-03 @ (4, 5, 35) Rank#14, time step = 0, continuity = 8.434087e-03 @ (4, 6, 35) Rank#44, time step = 0, continuity = 1.167705e-02 @ (4, 4, 5) Rank#45, time step = 0, continuity = 8.865762e-03 @ (4, 5, 4) Rank#46, time step = 0, continuity = 8.451129e-03 @ (4, 6, 4) Rank#25, time step = 0, continuity = 5.328321e-03 @ (41, 4, 4) Rank#28, time step = 0, continuity = 1.454749e-02 @ (4, 4, 20) Rank#33, time step = 0, continuity = 4.844833e-03 @ (41, 51, 4) Rank#26, time step = 0, continuity = 1.016585e+00 @ (39, 4, 17) Rank#40, time step = 0, continuity = 2.881710e-04 @ (41, 5, 4) Rank#29, time step = 0, continuity = 9.027575e-03 @ (4, 5, 20) Rank#32, time step = 0, continuity = 2.879931e-04 @ (41, 51, 4) Rank#36, time step = 0, continuity = 1.161600e-02 @ (4, 51, 5) Rank#41, time step = 0, continuity = 4.847584e-03 @ (41, 5, 4) Rank#30, time step = 0, continuity = 8.459037e-03 @ (4, 6, 22) Rank#37, time step = 0, continuity = 8.865921e-03 @ (4, 51, 4) Rank#42, time step = 0, continuity = 1.182950e-01 @ (40, 5, 4) Rank#31, time step = 0, continuity = 8.378688e-03 @ (4, 12, 28) Rank#38, time step = 0, continuity = 8.450654e-03 @ (4, 51, 4) Rank#43, time step = 0, continuity = 1.077929e-01 @ (4, 5, 4) Rank#39, time step = 0, continuity = 8.377602e-03 @ (4, 51, 5) Rank#35, time step = 0, continuity = 1.076056e-01 @ (4, 51, 4) Rank#18, time step = 0, continuity = 9.940864e-02 @ (40, 51, 4) Rank#9, time step = 0, continuity = 2.400260e-04 @ (4, 4, 35) Rank#23, time step = 0, continuity = 8.378040e-03 @ (4, 51, 29) Rank#22, time step = 0, continuity = 8.458582e-03 @ (4, 51, 22) Rank#47, time step = 0, continuity = 8.378259e-03 @ (4, 12, 5) Rank#3, time step = 0, continuity = 3.642321e-03 @ (4, 4, 4) Rank#7, time step = 0, continuity = 8.373002e-03 @ (4, 51, 35) Rank#15, time step = 0, continuity = 8.373655e-03 @ (4, 12, 35) Rank#19, time step = 0, continuity = 4.094089e+00 @ (4, 45, 17) Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 0 KSP unpreconditioned resid norm 7.406650929703e+00 true resid norm 7.406650929703e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.561334348759e-01 true resid norm 5.561334348760e-01 ||r(i)||/||b|| 7.508568179523e-02 2 KSP unpreconditioned resid norm 1.554698924650e-01 true resid norm 1.554698924650e-01 ||r(i)||/||b|| 2.099057913497e-02 3 KSP unpreconditioned resid norm 5.829927905115e-02 true resid norm 5.829927905115e-02 ||r(i)||/||b|| 7.871206514857e-03 4 KSP unpreconditioned resid norm 3.294371415430e-02 true resid norm 3.294371415430e-02 ||r(i)||/||b|| 4.447855645821e-03 5 KSP unpreconditioned resid norm 2.126154585895e-02 true resid norm 2.126154585896e-02 ||r(i)||/||b|| 2.870601849709e-03 6 KSP unpreconditioned resid norm 1.282632551143e-02 true resid norm 1.282632551143e-02 ||r(i)||/||b|| 1.731730796168e-03 7 KSP unpreconditioned resid norm 8.505640366786e-03 true resid norm 8.505640366788e-03 ||r(i)||/||b|| 1.148378727108e-03 8 KSP unpreconditioned resid norm 6.213706906443e-03 true resid norm 6.213706906443e-03 ||r(i)||/||b|| 8.389361082921e-04 9 KSP unpreconditioned resid norm 4.308546423382e-03 true resid norm 4.308546423384e-03 ||r(i)||/||b|| 5.817131743181e-04 10 KSP unpreconditioned resid norm 3.252806301386e-03 true resid norm 3.252806301386e-03 ||r(i)||/||b|| 4.391737010774e-04 11 KSP unpreconditioned resid norm 2.550371639217e-03 true resid norm 2.550371639217e-03 ||r(i)||/||b|| 3.443353363649e-04 12 KSP unpreconditioned resid norm 1.973664425127e-03 true resid norm 1.973664425127e-03 ||r(i)||/||b|| 2.664719107002e-04 13 KSP unpreconditioned resid norm 1.548366484711e-03 true resid norm 1.548366484710e-03 ||r(i)||/||b|| 2.090508246448e-04 14 KSP unpreconditioned resid norm 1.253035127943e-03 true resid norm 1.253035127943e-03 ||r(i)||/||b|| 1.691770193891e-04 15 KSP unpreconditioned resid norm 9.627823525618e-04 true resid norm 9.627823525611e-04 ||r(i)||/||b|| 1.299888926451e-04 16 KSP unpreconditioned resid norm 7.830570268883e-04 true resid norm 7.830570268888e-04 ||r(i)||/||b|| 1.057234955881e-04 17 KSP unpreconditioned resid norm 6.258324642484e-04 true resid norm 6.258324642482e-04 ||r(i)||/||b|| 8.449601178563e-05 18 KSP unpreconditioned resid norm 5.131714996563e-04 true resid norm 5.131714996560e-04 ||r(i)||/||b|| 6.928522817216e-05 19 KSP unpreconditioned resid norm 4.157415220129e-04 true resid norm 4.157415220130e-04 ||r(i)||/||b|| 5.613083780495e-05 20 KSP unpreconditioned resid norm 3.503809296735e-04 true resid norm 3.503809296738e-04 ||r(i)||/||b|| 4.730625663330e-05 21 KSP unpreconditioned resid norm 2.855345717794e-04 true resid norm 2.855345717803e-04 ||r(i)||/||b|| 3.855110420220e-05 22 KSP unpreconditioned resid norm 2.384533498881e-04 true resid norm 2.384533498885e-04 ||r(i)||/||b|| 3.219449008084e-05 23 KSP unpreconditioned resid norm 1.934365375858e-04 true resid norm 1.934365375864e-04 ||r(i)||/||b|| 2.611659971859e-05 24 KSP unpreconditioned resid norm 1.510009889378e-04 true resid norm 1.510009889395e-04 ||r(i)||/||b|| 2.038721554083e-05 25 KSP unpreconditioned resid norm 1.178370506710e-04 true resid norm 1.178370506724e-04 ||r(i)||/||b|| 1.590962660328e-05 26 KSP unpreconditioned resid norm 8.921055829265e-05 true resid norm 8.921055829443e-05 ||r(i)||/||b|| 1.204465542404e-05 27 KSP unpreconditioned resid norm 6.903213040845e-05 true resid norm 6.903213041012e-05 ||r(i)||/||b|| 9.320289435172e-06 28 KSP unpreconditioned resid norm 5.357460942945e-05 true resid norm 5.357460943198e-05 ||r(i)||/||b|| 7.233310971512e-06 29 KSP unpreconditioned resid norm 4.182989256886e-05 true resid norm 4.182989257160e-05 ||r(i)||/||b|| 5.647612256688e-06 30 KSP unpreconditioned resid norm 3.087441942016e-05 true resid norm 3.087441942016e-05 ||r(i)||/||b|| 4.168472324833e-06 31 KSP unpreconditioned resid norm 2.643983394241e-05 true resid norm 2.643983394242e-05 ||r(i)||/||b|| 3.569742140323e-06 32 KSP unpreconditioned resid norm 2.216484802712e-05 true resid norm 2.216484802713e-05 ||r(i)||/||b|| 2.992560097336e-06 33 KSP unpreconditioned resid norm 1.907523007213e-05 true resid norm 1.907523007214e-05 ||r(i)||/||b|| 2.575419073098e-06 34 KSP unpreconditioned resid norm 1.547185278986e-05 true resid norm 1.547185278987e-05 ||r(i)||/||b|| 2.088913455854e-06 35 KSP unpreconditioned resid norm 1.256379017347e-05 true resid norm 1.256379017349e-05 ||r(i)||/||b|| 1.696284905652e-06 36 KSP unpreconditioned resid norm 1.072584068780e-05 true resid norm 1.072584068780e-05 ||r(i)||/||b|| 1.448136383042e-06 37 KSP unpreconditioned resid norm 9.211701529799e-06 true resid norm 9.211701529800e-06 ||r(i)||/||b|| 1.243706719438e-06 38 KSP unpreconditioned resid norm 8.028399995262e-06 true resid norm 8.028399995246e-06 ||r(i)||/||b|| 1.083944696658e-06 39 KSP unpreconditioned resid norm 6.857325667061e-06 true resid norm 6.857325667056e-06 ||r(i)||/||b|| 9.258335153282e-07 40 KSP unpreconditioned resid norm 5.759722349850e-06 true resid norm 5.759722349846e-06 ||r(i)||/||b|| 7.776419335152e-07 41 KSP unpreconditioned resid norm 4.722558786184e-06 true resid norm 4.722558786179e-06 ||r(i)||/||b|| 6.376105517867e-07 42 KSP unpreconditioned resid norm 3.797562376720e-06 true resid norm 3.797562376716e-06 ||r(i)||/||b|| 5.127232824604e-07 43 KSP unpreconditioned resid norm 2.942658206923e-06 true resid norm 2.942658206913e-06 ||r(i)||/||b|| 3.972994319352e-07 44 KSP unpreconditioned resid norm 2.251149576887e-06 true resid norm 2.251149576873e-06 ||r(i)||/||b|| 3.039362322107e-07 45 KSP unpreconditioned resid norm 1.698940938700e-06 true resid norm 1.698940938710e-06 ||r(i)||/||b|| 2.293804520876e-07 46 KSP unpreconditioned resid norm 1.261300739337e-06 true resid norm 1.261300739358e-06 ||r(i)||/||b|| 1.702929908983e-07 47 KSP unpreconditioned resid norm 9.471923629982e-07 true resid norm 9.471923630326e-07 ||r(i)||/||b|| 1.278840291007e-07 48 KSP unpreconditioned resid norm 7.128407378936e-07 true resid norm 7.128407379083e-07 ||r(i)||/||b|| 9.624332841847e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 7.12841e-07 0 KSP unpreconditioned resid norm 2.178065518847e+00 true resid norm 2.178065518847e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.460392868474e-01 true resid norm 1.460392868474e-01 ||r(i)||/||b|| 6.704999715746e-02 2 KSP unpreconditioned resid norm 3.941062007261e-02 true resid norm 3.941062007261e-02 ||r(i)||/||b|| 1.809432256817e-02 3 KSP unpreconditioned resid norm 1.469001086082e-02 true resid norm 1.469001086082e-02 ||r(i)||/||b|| 6.744522023654e-03 4 KSP unpreconditioned resid norm 7.045501244561e-03 true resid norm 7.045501244561e-03 ||r(i)||/||b|| 3.234751748098e-03 5 KSP unpreconditioned resid norm 4.607801857748e-03 true resid norm 4.607801857748e-03 ||r(i)||/||b|| 2.115547864781e-03 6 KSP unpreconditioned resid norm 2.920588014185e-03 true resid norm 2.920588014185e-03 ||r(i)||/||b|| 1.340909164078e-03 7 KSP unpreconditioned resid norm 1.837118592260e-03 true resid norm 1.837118592261e-03 ||r(i)||/||b|| 8.434634203445e-04 8 KSP unpreconditioned resid norm 1.326010801107e-03 true resid norm 1.326010801107e-03 ||r(i)||/||b|| 6.088020721291e-04 9 KSP unpreconditioned resid norm 9.748366701558e-04 true resid norm 9.748366701566e-04 ||r(i)||/||b|| 4.475699476079e-04 10 KSP unpreconditioned resid norm 7.331750895867e-04 true resid norm 7.331750895872e-04 ||r(i)||/||b|| 3.366175550015e-04 11 KSP unpreconditioned resid norm 5.814568566459e-04 true resid norm 5.814568566461e-04 ||r(i)||/||b|| 2.669602230120e-04 12 KSP unpreconditioned resid norm 5.038353768751e-04 true resid norm 5.038353768752e-04 ||r(i)||/||b|| 2.313224154717e-04 13 KSP unpreconditioned resid norm 4.261654180571e-04 true resid norm 4.261654180568e-04 ||r(i)||/||b|| 1.956623500851e-04 14 KSP unpreconditioned resid norm 3.777823804727e-04 true resid norm 3.777823804721e-04 ||r(i)||/||b|| 1.734485841694e-04 15 KSP unpreconditioned resid norm 3.149422832642e-04 true resid norm 3.149422832633e-04 ||r(i)||/||b|| 1.445972495034e-04 16 KSP unpreconditioned resid norm 2.738474743364e-04 true resid norm 2.738474743356e-04 ||r(i)||/||b|| 1.257296770763e-04 17 KSP unpreconditioned resid norm 2.130851183760e-04 true resid norm 2.130851183749e-04 ||r(i)||/||b|| 9.783228122895e-05 18 KSP unpreconditioned resid norm 1.744002705546e-04 true resid norm 1.744002705535e-04 ||r(i)||/||b|| 8.007117740235e-05 19 KSP unpreconditioned resid norm 1.268436039537e-04 true resid norm 1.268436039527e-04 ||r(i)||/||b|| 5.823681742132e-05 20 KSP unpreconditioned resid norm 1.025539525209e-04 true resid norm 1.025539525202e-04 ||r(i)||/||b|| 4.708487951018e-05 21 KSP unpreconditioned resid norm 8.111346657473e-05 true resid norm 8.111346657392e-05 ||r(i)||/||b|| 3.724105903705e-05 22 KSP unpreconditioned resid norm 6.811645792110e-05 true resid norm 6.811645792094e-05 ||r(i)||/||b|| 3.127383328533e-05 23 KSP unpreconditioned resid norm 5.476915383296e-05 true resid norm 5.476915383309e-05 ||r(i)||/||b|| 2.514577883869e-05 24 KSP unpreconditioned resid norm 4.361783613715e-05 true resid norm 4.361783613789e-05 ||r(i)||/||b|| 2.002595227759e-05 25 KSP unpreconditioned resid norm 3.473686153395e-05 true resid norm 3.473686153498e-05 ||r(i)||/||b|| 1.594849247390e-05 26 KSP unpreconditioned resid norm 2.662103935785e-05 true resid norm 2.662103936019e-05 ||r(i)||/||b|| 1.222233175717e-05 27 KSP unpreconditioned resid norm 2.200010814357e-05 true resid norm 2.200010814643e-05 ||r(i)||/||b|| 1.010075590292e-05 28 KSP unpreconditioned resid norm 1.774330318901e-05 true resid norm 1.774330319304e-05 ||r(i)||/||b|| 8.146358793851e-06 29 KSP unpreconditioned resid norm 1.494333750491e-05 true resid norm 1.494333750969e-05 ||r(i)||/||b|| 6.860830117544e-06 30 KSP unpreconditioned resid norm 1.206656267301e-05 true resid norm 1.206656267301e-05 ||r(i)||/||b|| 5.540036591461e-06 31 KSP unpreconditioned resid norm 1.120278995565e-05 true resid norm 1.120278995564e-05 ||r(i)||/||b|| 5.143458660314e-06 32 KSP unpreconditioned resid norm 9.261235878008e-06 true resid norm 9.261235878003e-06 ||r(i)||/||b|| 4.252046505427e-06 33 KSP unpreconditioned resid norm 8.197470936141e-06 true resid norm 8.197470936131e-06 ||r(i)||/||b|| 3.763647541913e-06 34 KSP unpreconditioned resid norm 6.820318077282e-06 true resid norm 6.820318077276e-06 ||r(i)||/||b|| 3.131364974221e-06 35 KSP unpreconditioned resid norm 5.688691374400e-06 true resid norm 5.688691374394e-06 ||r(i)||/||b|| 2.611809114634e-06 36 KSP unpreconditioned resid norm 4.595604182694e-06 true resid norm 4.595604182687e-06 ||r(i)||/||b|| 2.109947631475e-06 37 KSP unpreconditioned resid norm 3.771797432721e-06 true resid norm 3.771797432722e-06 ||r(i)||/||b|| 1.731718995634e-06 38 KSP unpreconditioned resid norm 3.109066412989e-06 true resid norm 3.109066412982e-06 ||r(i)||/||b|| 1.427443934115e-06 39 KSP unpreconditioned resid norm 2.669628603745e-06 true resid norm 2.669628603739e-06 ||r(i)||/||b|| 1.225687923820e-06 40 KSP unpreconditioned resid norm 2.210365942259e-06 true resid norm 2.210365942256e-06 ||r(i)||/||b|| 1.014829867664e-06 41 KSP unpreconditioned resid norm 1.821949773643e-06 true resid norm 1.821949773631e-06 ||r(i)||/||b|| 8.364990666563e-07 42 KSP unpreconditioned resid norm 1.446964504975e-06 true resid norm 1.446964504973e-06 ||r(i)||/||b|| 6.643347008862e-07 43 KSP unpreconditioned resid norm 1.161968636199e-06 true resid norm 1.161968636218e-06 ||r(i)||/||b|| 5.334865393916e-07 44 KSP unpreconditioned resid norm 9.487319749258e-07 true resid norm 9.487319749359e-07 ||r(i)||/||b|| 4.355846813269e-07 45 KSP unpreconditioned resid norm 7.697968173503e-07 true resid norm 7.697968173499e-07 ||r(i)||/||b|| 3.534314329339e-07 46 KSP unpreconditioned resid norm 6.509475897671e-07 true resid norm 6.509475897554e-07 ||r(i)||/||b|| 2.988650176603e-07 47 KSP unpreconditioned resid norm 5.418206977645e-07 true resid norm 5.418206977467e-07 ||r(i)||/||b|| 2.487623503785e-07 48 KSP unpreconditioned resid norm 4.629579528470e-07 true resid norm 4.629579528207e-07 ||r(i)||/||b|| 2.125546494422e-07 49 KSP unpreconditioned resid norm 3.742046372363e-07 true resid norm 3.742046372009e-07 ||r(i)||/||b|| 1.718059599048e-07 50 KSP unpreconditioned resid norm 2.906109875748e-07 true resid norm 2.906109875442e-07 ||r(i)||/||b|| 1.334261917420e-07 51 KSP unpreconditioned resid norm 2.167356702683e-07 true resid norm 2.167356702358e-07 ||r(i)||/||b|| 9.950833359255e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 2.16736e-07 Rank#2, time step = 1, continuity = 2.025209e-03 @ (4, 51, 35) Rank#44, time step = 1, continuity = 8.147067e-03 @ (5, 5, 4) Rank#24, time step = 1, continuity = 3.314627e-04 @ (41, 4, 4) Rank#10, time step = 1, continuity = 2.015249e-03 @ (4, 5, 35) Rank#1, time step = 1, continuity = 9.967610e-05 @ (4, 4, 4) Rank#46, time step = 1, continuity = 2.260864e-03 @ (4, 50, 34) Rank#27, time step = 1, continuity = 4.478382e+00 @ (4, 11, 17) Rank#11, time step = 1, continuity = 3.502620e-02 @ (4, 4, 35) Rank#31, time step = 1, continuity = 2.189822e-03 @ (4, 50, 5) Rank#0, time step = 1, continuity = 3.535561e-08 @ (19, 51, 7) Rank#47, time step = 1, continuity = 2.189827e-03 @ (4, 50, 34) Rank#8, time step = 1, continuity = 3.448571e-05 @ (41, 4, 35) Rank#30, time step = 1, continuity = 2.226857e-03 @ (4, 50, 5) Rank#4, time step = 1, continuity = 4.905157e-03 @ (4, 4, 4) Rank#28, time step = 1, continuity = 1.385920e-02 @ (4, 4, 20) Rank#5, time step = 1, continuity = 2.806331e-03 @ (4, 4, 4) Rank#25, time step = 1, continuity = 4.553406e-03 @ (41, 4, 4) Rank#29, time step = 1, continuity = 2.459065e-03 @ (4, 50, 5) Rank#3, time step = 1, continuity = 3.336243e-02 @ (4, 51, 35) Rank#6, time step = 1, continuity = 2.297121e-03 @ (4, 4, 4) Rank#26, time step = 1, continuity = 9.589653e-01 @ (39, 4, 17) Rank#20, time step = 1, continuity = 1.330094e-02 @ (4, 51, 20) Rank#21, time step = 1, continuity = 2.511545e-03 @ (4, 4, 5) Rank#16, time step = 1, continuity = 2.985854e-04 @ (41, 51, 4) Rank#37, time step = 1, continuity = 2.769053e-03 @ (4, 4, 34) Rank#40, time step = 1, continuity = 2.449979e-04 @ (41, 5, 4) Rank#12, time step = 1, continuity = 4.912665e-03 @ (4, 50, 4) Rank#22, time step = 1, continuity = 2.246064e-03 @ (4, 4, 5) Rank#17, time step = 1, continuity = 4.362478e-03 @ (41, 51, 4) Rank#36, time step = 1, continuity = 8.123727e-03 @ (5, 51, 4) Rank#41, time step = 1, continuity = 4.216866e-03 @ (41, 5, 4) Rank#14, time step = 1, continuity = 2.279742e-03 @ (4, 50, 4) Rank#23, time step = 1, continuity = 2.197173e-03 @ (4, 4, 5) Rank#18, time step = 1, continuity = 8.029293e-02 @ (39, 51, 4) Rank#38, time step = 1, continuity = 2.278785e-03 @ (4, 4, 34) Rank#42, time step = 1, continuity = 9.597068e-02 @ (40, 5, 4) Rank#15, time step = 1, continuity = 2.197458e-03 @ (4, 50, 4) Rank#19, time step = 1, continuity = 4.478201e+00 @ (4, 45, 17) Rank#43, time step = 1, continuity = 8.285460e-02 @ (19, 5, 4) Rank#39, time step = 1, continuity = 2.197159e-03 @ (4, 4, 34) Rank#45, time step = 1, continuity = 2.732214e-03 @ (4, 50, 34) Rank#32, time step = 1, continuity = 2.455089e-04 @ (41, 51, 4) Rank#33, time step = 1, continuity = 4.212290e-03 @ (41, 51, 4) Rank#34, time step = 1, continuity = 9.554038e-02 @ (40, 51, 4) Rank#9, time step = 1, continuity = 1.731917e-04 @ (41, 4, 35) Rank#13, time step = 1, continuity = 2.771882e-03 @ (4, 50, 4) Rank#7, time step = 1, continuity = 2.204604e-03 @ (4, 4, 4) Rank#35, time step = 1, continuity = 8.228460e-02 @ (19, 51, 4) Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 0 KSP unpreconditioned resid norm 1.145993263262e+01 true resid norm 1.145993263262e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 8.467018295731e-01 true resid norm 8.467018295732e-01 ||r(i)||/||b|| 7.388366552551e-02 2 KSP unpreconditioned resid norm 2.353082733202e-01 true resid norm 2.353082733202e-01 ||r(i)||/||b|| 2.053312884671e-02 3 KSP unpreconditioned resid norm 8.280650623931e-02 true resid norm 8.280650623932e-02 ||r(i)||/||b|| 7.225741101098e-03 4 KSP unpreconditioned resid norm 4.146387000857e-02 true resid norm 4.146387000857e-02 ||r(i)||/||b|| 3.618160013485e-03 5 KSP unpreconditioned resid norm 2.760201858697e-02 true resid norm 2.760201858697e-02 ||r(i)||/||b|| 2.408567264035e-03 6 KSP unpreconditioned resid norm 1.750509619293e-02 true resid norm 1.750509619293e-02 ||r(i)||/||b|| 1.527504284197e-03 7 KSP unpreconditioned resid norm 1.101437000383e-02 true resid norm 1.101437000383e-02 ||r(i)||/||b|| 9.611199609047e-04 8 KSP unpreconditioned resid norm 8.020603798473e-03 true resid norm 8.020603798472e-03 ||r(i)||/||b|| 6.998822816501e-04 9 KSP unpreconditioned resid norm 5.769911324715e-03 true resid norm 5.769911324718e-03 ||r(i)||/||b|| 5.034856233182e-04 10 KSP unpreconditioned resid norm 4.217969701212e-03 true resid norm 4.217969701212e-03 ||r(i)||/||b|| 3.680623469991e-04 11 KSP unpreconditioned resid norm 3.290098525645e-03 true resid norm 3.290098525644e-03 ||r(i)||/||b|| 2.870957998723e-04 12 KSP unpreconditioned resid norm 2.609406224541e-03 true resid norm 2.609406224541e-03 ||r(i)||/||b|| 2.276982167517e-04 13 KSP unpreconditioned resid norm 2.004708786049e-03 true resid norm 2.004708786048e-03 ||r(i)||/||b|| 1.749319869772e-04 14 KSP unpreconditioned resid norm 1.621432183334e-03 true resid norm 1.621432183333e-03 ||r(i)||/||b|| 1.414870606410e-04 15 KSP unpreconditioned resid norm 1.273824611796e-03 true resid norm 1.273824611795e-03 ||r(i)||/||b|| 1.111546335071e-04 16 KSP unpreconditioned resid norm 1.022867010282e-03 true resid norm 1.022867010282e-03 ||r(i)||/||b|| 8.925593570862e-05 17 KSP unpreconditioned resid norm 8.088159574836e-04 true resid norm 8.088159574832e-04 ||r(i)||/||b|| 7.057772357066e-05 18 KSP unpreconditioned resid norm 6.782535622143e-04 true resid norm 6.782535622147e-04 ||r(i)||/||b|| 5.918477742916e-05 19 KSP unpreconditioned resid norm 5.390023303178e-04 true resid norm 5.390023303189e-04 ||r(i)||/||b|| 4.703363864328e-05 20 KSP unpreconditioned resid norm 4.544503585906e-04 true resid norm 4.544503585916e-04 ||r(i)||/||b|| 3.965558726743e-05 21 KSP unpreconditioned resid norm 3.748239986292e-04 true resid norm 3.748239986313e-04 ||r(i)||/||b|| 3.270734747292e-05 22 KSP unpreconditioned resid norm 3.106950604455e-04 true resid norm 3.106950604470e-04 ||r(i)||/||b|| 2.711142119306e-05 23 KSP unpreconditioned resid norm 2.519104118885e-04 true resid norm 2.519104118907e-04 ||r(i)||/||b|| 2.198184055408e-05 24 KSP unpreconditioned resid norm 1.993796457577e-04 true resid norm 1.993796457610e-04 ||r(i)||/||b|| 1.739797712192e-05 25 KSP unpreconditioned resid norm 1.536074482824e-04 true resid norm 1.536074482846e-04 ||r(i)||/||b|| 1.340387009321e-05 26 KSP unpreconditioned resid norm 1.160166482479e-04 true resid norm 1.160166482505e-04 ||r(i)||/||b|| 1.012367628762e-05 27 KSP unpreconditioned resid norm 9.118828433660e-05 true resid norm 9.118828433948e-05 ||r(i)||/||b|| 7.957139650188e-06 28 KSP unpreconditioned resid norm 6.968237176481e-05 true resid norm 6.968237176823e-05 ||r(i)||/||b|| 6.080521936918e-06 29 KSP unpreconditioned resid norm 5.489756729503e-05 true resid norm 5.489756729856e-05 ||r(i)||/||b|| 4.790391798841e-06 30 KSP unpreconditioned resid norm 4.088864972904e-05 true resid norm 4.088864972904e-05 ||r(i)||/||b|| 3.567965976751e-06 31 KSP unpreconditioned resid norm 3.550536673258e-05 true resid norm 3.550536673258e-05 ||r(i)||/||b|| 3.098217753177e-06 32 KSP unpreconditioned resid norm 2.932517724995e-05 true resid norm 2.932517724995e-05 ||r(i)||/||b|| 2.558931033023e-06 33 KSP unpreconditioned resid norm 2.538035496051e-05 true resid norm 2.538035496053e-05 ||r(i)||/||b|| 2.214703678824e-06 34 KSP unpreconditioned resid norm 2.077966563835e-05 true resid norm 2.077966563835e-05 ||r(i)||/||b|| 1.813245008021e-06 35 KSP unpreconditioned resid norm 1.677227950577e-05 true resid norm 1.677227950575e-05 ||r(i)||/||b|| 1.463558298590e-06 36 KSP unpreconditioned resid norm 1.426971881665e-05 true resid norm 1.426971881664e-05 ||r(i)||/||b|| 1.245183481797e-06 37 KSP unpreconditioned resid norm 1.232421452779e-05 true resid norm 1.232421452778e-05 ||r(i)||/||b|| 1.075417711681e-06 38 KSP unpreconditioned resid norm 1.072305282161e-05 true resid norm 1.072305282161e-05 ||r(i)||/||b|| 9.356994639823e-07 39 KSP unpreconditioned resid norm 9.187459115632e-06 true resid norm 9.187459115628e-06 ||r(i)||/||b|| 8.017027159025e-07 40 KSP unpreconditioned resid norm 7.725611104268e-06 true resid norm 7.725611104266e-06 ||r(i)||/||b|| 6.741410575380e-07 41 KSP unpreconditioned resid norm 6.336099247960e-06 true resid norm 6.336099247978e-06 ||r(i)||/||b|| 5.528914916955e-07 42 KSP unpreconditioned resid norm 5.072527467147e-06 true resid norm 5.072527467184e-06 ||r(i)||/||b|| 4.426315258387e-07 43 KSP unpreconditioned resid norm 3.980186570456e-06 true resid norm 3.980186570487e-06 ||r(i)||/||b|| 3.473132607393e-07 44 KSP unpreconditioned resid norm 3.031763132156e-06 true resid norm 3.031763132231e-06 ||r(i)||/||b|| 2.645533119106e-07 45 KSP unpreconditioned resid norm 2.268918039035e-06 true resid norm 2.268918039109e-06 ||r(i)||/||b|| 1.979870311498e-07 46 KSP unpreconditioned resid norm 1.714692473145e-06 true resid norm 1.714692473173e-06 ||r(i)||/||b|| 1.496250046263e-07 47 KSP unpreconditioned resid norm 1.278527011803e-06 true resid norm 1.278527011797e-06 ||r(i)||/||b|| 1.115649675076e-07 48 KSP unpreconditioned resid norm 9.599919523504e-07 true resid norm 9.599919523332e-07 ||r(i)||/||b|| 8.376942370506e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 9.59992e-07 0 KSP unpreconditioned resid norm 1.903872986821e+00 true resid norm 1.903872986821e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.250166965029e-01 true resid norm 1.250166965029e-01 ||r(i)||/||b|| 6.566441005689e-02 2 KSP unpreconditioned resid norm 3.413148600032e-02 true resid norm 3.413148600032e-02 ||r(i)||/||b|| 1.792739654199e-02 3 KSP unpreconditioned resid norm 1.224574003913e-02 true resid norm 1.224574003913e-02 ||r(i)||/||b|| 6.432015225750e-03 4 KSP unpreconditioned resid norm 5.300073743184e-03 true resid norm 5.300073743185e-03 ||r(i)||/||b|| 2.783837881976e-03 5 KSP unpreconditioned resid norm 3.365317369016e-03 true resid norm 3.365317369016e-03 ||r(i)||/||b|| 1.767616533409e-03 6 KSP unpreconditioned resid norm 2.233194123812e-03 true resid norm 2.233194123811e-03 ||r(i)||/||b|| 1.172974320908e-03 7 KSP unpreconditioned resid norm 1.414030484365e-03 true resid norm 1.414030484365e-03 ||r(i)||/||b|| 7.427126148400e-04 8 KSP unpreconditioned resid norm 1.008018647560e-03 true resid norm 1.008018647560e-03 ||r(i)||/||b|| 5.294568779209e-04 9 KSP unpreconditioned resid norm 7.859431037379e-04 true resid norm 7.859431037391e-04 ||r(i)||/||b|| 4.128127817241e-04 10 KSP unpreconditioned resid norm 6.142350651412e-04 true resid norm 6.142350651422e-04 ||r(i)||/||b|| 3.226239719739e-04 11 KSP unpreconditioned resid norm 5.010222708685e-04 true resid norm 5.010222708691e-04 ||r(i)||/||b|| 2.631595039886e-04 12 KSP unpreconditioned resid norm 4.370374667634e-04 true resid norm 4.370374667636e-04 ||r(i)||/||b|| 2.295517977243e-04 13 KSP unpreconditioned resid norm 3.751450978703e-04 true resid norm 3.751450978698e-04 ||r(i)||/||b|| 1.970431328490e-04 14 KSP unpreconditioned resid norm 3.234752378984e-04 true resid norm 3.234752378973e-04 ||r(i)||/||b|| 1.699037909232e-04 15 KSP unpreconditioned resid norm 2.685396778927e-04 true resid norm 2.685396778914e-04 ||r(i)||/||b|| 1.410491559837e-04 16 KSP unpreconditioned resid norm 2.309017226947e-04 true resid norm 2.309017226933e-04 ||r(i)||/||b|| 1.212800035988e-04 17 KSP unpreconditioned resid norm 1.762487331106e-04 true resid norm 1.762487331090e-04 ||r(i)||/||b|| 9.257378739494e-05 18 KSP unpreconditioned resid norm 1.448381331042e-04 true resid norm 1.448381331027e-04 ||r(i)||/||b|| 7.607552294995e-05 19 KSP unpreconditioned resid norm 1.077151882122e-04 true resid norm 1.077151882107e-04 ||r(i)||/||b|| 5.657687721623e-05 20 KSP unpreconditioned resid norm 8.753358844749e-05 true resid norm 8.753358844627e-05 ||r(i)||/||b|| 4.597659037771e-05 21 KSP unpreconditioned resid norm 6.789005163891e-05 true resid norm 6.789005163755e-05 ||r(i)||/||b|| 3.565891848221e-05 22 KSP unpreconditioned resid norm 5.730497317295e-05 true resid norm 5.730497317199e-05 ||r(i)||/||b|| 3.009915764794e-05 23 KSP unpreconditioned resid norm 4.488629127859e-05 true resid norm 4.488629127750e-05 ||r(i)||/||b|| 2.357630555620e-05 24 KSP unpreconditioned resid norm 3.549059931238e-05 true resid norm 3.549059931175e-05 ||r(i)||/||b|| 1.864126417960e-05 25 KSP unpreconditioned resid norm 2.825201934536e-05 true resid norm 2.825201934469e-05 ||r(i)||/||b|| 1.483923535879e-05 26 KSP unpreconditioned resid norm 2.204358958226e-05 true resid norm 2.204358958227e-05 ||r(i)||/||b|| 1.157828790831e-05 27 KSP unpreconditioned resid norm 1.826364573014e-05 true resid norm 1.826364573052e-05 ||r(i)||/||b|| 9.592890837228e-06 28 KSP unpreconditioned resid norm 1.488023103673e-05 true resid norm 1.488023103794e-05 ||r(i)||/||b|| 7.815768772885e-06 29 KSP unpreconditioned resid norm 1.283432364115e-05 true resid norm 1.283432364331e-05 ||r(i)||/||b|| 6.741165892973e-06 30 KSP unpreconditioned resid norm 1.014605282698e-05 true resid norm 1.014605282698e-05 ||r(i)||/||b|| 5.329164758998e-06 31 KSP unpreconditioned resid norm 9.407914973543e-06 true resid norm 9.407914973543e-06 ||r(i)||/||b|| 4.941461451822e-06 32 KSP unpreconditioned resid norm 7.757298584563e-06 true resid norm 7.757298584557e-06 ||r(i)||/||b|| 4.074483244551e-06 33 KSP unpreconditioned resid norm 6.931504610008e-06 true resid norm 6.931504610005e-06 ||r(i)||/||b|| 3.640738987309e-06 34 KSP unpreconditioned resid norm 5.698043511339e-06 true resid norm 5.698043511334e-06 ||r(i)||/||b|| 2.992869561560e-06 35 KSP unpreconditioned resid norm 4.862023768014e-06 true resid norm 4.862023768017e-06 ||r(i)||/||b|| 2.553754269152e-06 36 KSP unpreconditioned resid norm 3.935139896939e-06 true resid norm 3.935139896942e-06 ||r(i)||/||b|| 2.066913036837e-06 37 KSP unpreconditioned resid norm 3.257659841958e-06 true resid norm 3.257659841967e-06 ||r(i)||/||b|| 1.711069942437e-06 38 KSP unpreconditioned resid norm 2.658817764230e-06 true resid norm 2.658817764249e-06 ||r(i)||/||b|| 1.396531062026e-06 39 KSP unpreconditioned resid norm 2.259760363675e-06 true resid norm 2.259760363695e-06 ||r(i)||/||b|| 1.186928108827e-06 40 KSP unpreconditioned resid norm 1.850320661460e-06 true resid norm 1.850320661493e-06 ||r(i)||/||b|| 9.718719023281e-07 41 KSP unpreconditioned resid norm 1.507757223827e-06 true resid norm 1.507757223866e-06 ||r(i)||/||b|| 7.919421276016e-07 42 KSP unpreconditioned resid norm 1.222830298328e-06 true resid norm 1.222830298355e-06 ||r(i)||/||b|| 6.422856497358e-07 43 KSP unpreconditioned resid norm 9.791313605986e-07 true resid norm 9.791313606237e-07 ||r(i)||/||b|| 5.142839713580e-07 44 KSP unpreconditioned resid norm 8.030348922423e-07 true resid norm 8.030348922612e-07 ||r(i)||/||b|| 4.217901602785e-07 45 KSP unpreconditioned resid norm 6.450601933616e-07 true resid norm 6.450601933832e-07 ||r(i)||/||b|| 3.388147202299e-07 46 KSP unpreconditioned resid norm 5.406035117950e-07 true resid norm 5.406035118192e-07 ||r(i)||/||b|| 2.839493577362e-07 47 KSP unpreconditioned resid norm 4.409627148245e-07 true resid norm 4.409627148583e-07 ||r(i)||/||b|| 2.316135151403e-07 48 KSP unpreconditioned resid norm 3.776631107440e-07 true resid norm 3.776631107863e-07 ||r(i)||/||b|| 1.983657068515e-07 49 KSP unpreconditioned resid norm 3.060662042891e-07 true resid norm 3.060662043298e-07 ||r(i)||/||b|| 1.607597809562e-07 50 KSP unpreconditioned resid norm 2.450411530613e-07 true resid norm 2.450411530946e-07 ||r(i)||/||b|| 1.287066704506e-07 51 KSP unpreconditioned resid norm 1.892344498531e-07 true resid norm 1.892344498921e-07 ||r(i)||/||b|| 9.939447179615e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 1.89234e-07 Rank#24, time step = 2, continuity = 3.572595e-04 @ (41, 4, 4) Rank#44, time step = 2, continuity = 1.147952e-02 @ (5, 5, 4) Rank#27, time step = 2, continuity = 3.980041e+00 @ (4, 11, 17) Rank#45, time step = 2, continuity = 2.594768e-03 @ (4, 50, 34) Rank#46, time step = 2, continuity = 2.060500e-03 @ (4, 50, 34) Rank#1, time step = 2, continuity = 2.221491e-04 @ (4, 4, 4) Rank#0, time step = 2, continuity = 2.750387e-08 @ (19, 51, 7) Rank#4, time step = 2, continuity = 6.838965e-03 @ (4, 51, 35) Rank#5, time step = 2, continuity = 2.685059e-03 @ (4, 4, 4) Rank#2, time step = 2, continuity = 3.305379e-03 @ (4, 51, 35) Rank#36, time step = 2, continuity = 1.145714e-02 @ (5, 51, 4) Rank#31, time step = 2, continuity = 2.043125e-03 @ (4, 50, 5) Rank#37, time step = 2, continuity = 2.641305e-03 @ (4, 4, 34) Rank#30, time step = 2, continuity = 2.021057e-03 @ (4, 50, 5) Rank#28, time step = 2, continuity = 2.169531e-02 @ (4, 4, 20) Rank#29, time step = 2, continuity = 2.251697e-03 @ (4, 50, 5) Rank#13, time step = 2, continuity = 2.641033e-03 @ (4, 50, 4) Rank#14, time step = 2, continuity = 2.080491e-03 @ (4, 50, 4) Rank#8, time step = 2, continuity = 3.839277e-05 @ (41, 4, 35) Rank#20, time step = 2, continuity = 2.088852e-02 @ (4, 51, 20) Rank#9, time step = 2, continuity = 2.025106e-04 @ (4, 50, 4) Rank#22, time step = 2, continuity = 2.042619e-03 @ (4, 4, 5) Rank#10, time step = 2, continuity = 3.293410e-03 @ (4, 5, 35) Rank#21, time step = 2, continuity = 2.318005e-03 @ (4, 4, 5) Rank#25, time step = 2, continuity = 5.028256e-03 @ (41, 4, 4) Rank#26, time step = 2, continuity = 8.660592e-01 @ (39, 4, 17) Rank#11, time step = 2, continuity = 4.892623e-02 @ (4, 4, 35) Rank#38, time step = 2, continuity = 2.080645e-03 @ (4, 4, 34) Rank#40, time step = 2, continuity = 2.570407e-04 @ (41, 5, 4) Rank#41, time step = 2, continuity = 4.631534e-03 @ (41, 5, 4) Rank#16, time step = 2, continuity = 3.189125e-04 @ (41, 51, 4) Rank#42, time step = 2, continuity = 1.119371e-01 @ (40, 5, 4) Rank#17, time step = 2, continuity = 4.813800e-03 @ (41, 51, 4) Rank#43, time step = 2, continuity = 1.055263e-01 @ (19, 5, 4) Rank#18, time step = 2, continuity = 9.377534e-02 @ (39, 51, 4) Rank#15, time step = 2, continuity = 2.050923e-03 @ (4, 50, 4) Rank#19, time step = 2, continuity = 4.025784e+00 @ (4, 51, 13) Rank#32, time step = 2, continuity = 2.570683e-04 @ (41, 51, 4) Rank#33, time step = 2, continuity = 4.597828e-03 @ (41, 51, 4) Rank#23, time step = 2, continuity = 2.050687e-03 @ (4, 4, 5) Rank#34, time step = 2, continuity = 1.111301e-01 @ (40, 51, 4) Rank#35, time step = 2, continuity = 1.047201e-01 @ (19, 51, 4) Rank#47, time step = 2, continuity = 2.043574e-03 @ (4, 50, 34) Rank#3, time step = 2, continuity = 4.698379e-02 @ (4, 51, 35) Rank#12, time step = 2, continuity = 7.182269e-03 @ (4, 4, 35) Rank#7, time step = 2, continuity = 2.058267e-03 @ (4, 4, 4) Rank#6, time step = 2, continuity = 2.100210e-03 @ (4, 4, 4) Rank#39, time step = 2, continuity = 2.051061e-03 @ (4, 5, 34) Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 0 KSP unpreconditioned resid norm 1.498434584652e+01 true resid norm 1.498434584652e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.092790363921e+00 true resid norm 1.092790363921e+00 ||r(i)||/||b|| 7.292880017017e-02 2 KSP unpreconditioned resid norm 3.035607510875e-01 true resid norm 3.035607510875e-01 ||r(i)||/||b|| 2.025852541024e-02 3 KSP unpreconditioned resid norm 1.046104233532e-01 true resid norm 1.046104233532e-01 ||r(i)||/||b|| 6.981313994265e-03 4 KSP unpreconditioned resid norm 4.900330197785e-02 true resid norm 4.900330197786e-02 ||r(i)||/||b|| 3.270299716771e-03 5 KSP unpreconditioned resid norm 3.243734826670e-02 true resid norm 3.243734826671e-02 ||r(i)||/||b|| 2.164749038694e-03 6 KSP unpreconditioned resid norm 2.125601515562e-02 true resid norm 2.125601515562e-02 ||r(i)||/||b|| 1.418548088341e-03 7 KSP unpreconditioned resid norm 1.320366086225e-02 true resid norm 1.320366086225e-02 ||r(i)||/||b|| 8.811636488838e-04 8 KSP unpreconditioned resid norm 9.434930974812e-03 true resid norm 9.434930974811e-03 ||r(i)||/||b|| 6.296525101231e-04 9 KSP unpreconditioned resid norm 6.928273956785e-03 true resid norm 6.928273956791e-03 ||r(i)||/||b|| 4.623674618670e-04 10 KSP unpreconditioned resid norm 5.054559978185e-03 true resid norm 5.054559978187e-03 ||r(i)||/||b|| 3.373226986323e-04 11 KSP unpreconditioned resid norm 3.879257749830e-03 true resid norm 3.879257749830e-03 ||r(i)||/||b|| 2.588873608208e-04 12 KSP unpreconditioned resid norm 3.115677995555e-03 true resid norm 3.115677995557e-03 ||r(i)||/||b|| 2.079288630595e-04 13 KSP unpreconditioned resid norm 2.397098909417e-03 true resid norm 2.397098909416e-03 ||r(i)||/||b|| 1.599735439885e-04 14 KSP unpreconditioned resid norm 1.919635663198e-03 true resid norm 1.919635663196e-03 ||r(i)||/||b|| 1.281094071679e-04 15 KSP unpreconditioned resid norm 1.516959045641e-03 true resid norm 1.516959045638e-03 ||r(i)||/||b|| 1.012362542333e-04 16 KSP unpreconditioned resid norm 1.227746545064e-03 true resid norm 1.227746545063e-03 ||r(i)||/||b|| 8.193527816555e-05 17 KSP unpreconditioned resid norm 9.567761331953e-04 true resid norm 9.567761331946e-04 ||r(i)||/||b|| 6.385171184614e-05 18 KSP unpreconditioned resid norm 8.061097903490e-04 true resid norm 8.061097903493e-04 ||r(i)||/||b|| 5.379679557628e-05 19 KSP unpreconditioned resid norm 6.448576549729e-04 true resid norm 6.448576549744e-04 ||r(i)||/||b|| 4.303542253894e-05 20 KSP unpreconditioned resid norm 5.367404070636e-04 true resid norm 5.367404070649e-04 ||r(i)||/||b|| 3.582007600216e-05 21 KSP unpreconditioned resid norm 4.465369943436e-04 true resid norm 4.465369943470e-04 ||r(i)||/||b|| 2.980023278433e-05 22 KSP unpreconditioned resid norm 3.708072163845e-04 true resid norm 3.708072163881e-04 ||r(i)||/||b|| 2.474630659131e-05 23 KSP unpreconditioned resid norm 2.994136346519e-04 true resid norm 2.994136346554e-04 ||r(i)||/||b|| 1.998176214846e-05 24 KSP unpreconditioned resid norm 2.368415263321e-04 true resid norm 2.368415263359e-04 ||r(i)||/||b|| 1.580593031967e-05 25 KSP unpreconditioned resid norm 1.843154328279e-04 true resid norm 1.843154328311e-04 ||r(i)||/||b|| 1.230053248363e-05 26 KSP unpreconditioned resid norm 1.374799193076e-04 true resid norm 1.374799193111e-04 ||r(i)||/||b|| 9.174902976697e-06 27 KSP unpreconditioned resid norm 1.078590988123e-04 true resid norm 1.078590988147e-04 ||r(i)||/||b|| 7.198118617891e-06 28 KSP unpreconditioned resid norm 8.366418218981e-05 true resid norm 8.366418219349e-05 ||r(i)||/||b|| 5.583439080388e-06 29 KSP unpreconditioned resid norm 6.486198779907e-05 true resid norm 6.486198780324e-05 ||r(i)||/||b|| 4.328649943589e-06 30 KSP unpreconditioned resid norm 4.869596253714e-05 true resid norm 4.869596253714e-05 ||r(i)||/||b|| 3.249789015545e-06 31 KSP unpreconditioned resid norm 4.197253762996e-05 true resid norm 4.197253762997e-05 ||r(i)||/||b|| 2.801092424047e-06 32 KSP unpreconditioned resid norm 3.477255859637e-05 true resid norm 3.477255859636e-05 ||r(i)||/||b|| 2.320592367030e-06 33 KSP unpreconditioned resid norm 2.984747219315e-05 true resid norm 2.984747219315e-05 ||r(i)||/||b|| 1.991910257470e-06 34 KSP unpreconditioned resid norm 2.473689623528e-05 true resid norm 2.473689623530e-05 ||r(i)||/||b|| 1.650849258865e-06 35 KSP unpreconditioned resid norm 1.955260027410e-05 true resid norm 1.955260027416e-05 ||r(i)||/||b|| 1.304868459020e-06 36 KSP unpreconditioned resid norm 1.694685253795e-05 true resid norm 1.694685253801e-05 ||r(i)||/||b|| 1.130970461547e-06 37 KSP unpreconditioned resid norm 1.449937027782e-05 true resid norm 1.449937027791e-05 ||r(i)||/||b|| 9.676345184785e-07 38 KSP unpreconditioned resid norm 1.264545410891e-05 true resid norm 1.264545410903e-05 ||r(i)||/||b|| 8.439109880776e-07 39 KSP unpreconditioned resid norm 1.079008846191e-05 true resid norm 1.079008846223e-05 ||r(i)||/||b|| 7.200907248641e-07 40 KSP unpreconditioned resid norm 9.100895988825e-06 true resid norm 9.100895989285e-06 ||r(i)||/||b|| 6.073602466538e-07 41 KSP unpreconditioned resid norm 7.434748605604e-06 true resid norm 7.434748606064e-06 ||r(i)||/||b|| 4.961677127729e-07 42 KSP unpreconditioned resid norm 5.974987079856e-06 true resid norm 5.974987080279e-06 ||r(i)||/||b|| 3.987486101482e-07 43 KSP unpreconditioned resid norm 4.731598282761e-06 true resid norm 4.731598283153e-06 ||r(i)||/||b|| 3.157694257473e-07 44 KSP unpreconditioned resid norm 3.527588850234e-06 true resid norm 3.527588850504e-06 ||r(i)||/||b|| 2.354182749542e-07 45 KSP unpreconditioned resid norm 2.688764015030e-06 true resid norm 2.688764015219e-06 ||r(i)||/||b|| 1.794381978873e-07 46 KSP unpreconditioned resid norm 2.023673546681e-06 true resid norm 2.023673546778e-06 ||r(i)||/||b|| 1.350525119686e-07 47 KSP unpreconditioned resid norm 1.510010378674e-06 true resid norm 1.510010378681e-06 ||r(i)||/||b|| 1.007725258178e-07 48 KSP unpreconditioned resid norm 1.140779932135e-06 true resid norm 1.140779932032e-06 ||r(i)||/||b|| 7.613144702591e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 1.14078e-06 0 KSP unpreconditioned resid norm 1.652982043410e+00 true resid norm 1.652982043410e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.040644076118e-01 true resid norm 1.040644076118e-01 ||r(i)||/||b|| 6.295555842646e-02 2 KSP unpreconditioned resid norm 2.849057515307e-02 true resid norm 2.849057515307e-02 ||r(i)||/||b|| 1.723586488229e-02 3 KSP unpreconditioned resid norm 1.053181684125e-02 true resid norm 1.053181684125e-02 ||r(i)||/||b|| 6.371404264939e-03 4 KSP unpreconditioned resid norm 4.749233649129e-03 true resid norm 4.749233649130e-03 ||r(i)||/||b|| 2.873130817158e-03 5 KSP unpreconditioned resid norm 3.048765576587e-03 true resid norm 3.048765576587e-03 ||r(i)||/||b|| 1.844403324732e-03 6 KSP unpreconditioned resid norm 1.972895444966e-03 true resid norm 1.972895444966e-03 ||r(i)||/||b|| 1.193537130565e-03 7 KSP unpreconditioned resid norm 1.216104509688e-03 true resid norm 1.216104509688e-03 ||r(i)||/||b|| 7.357034001285e-04 8 KSP unpreconditioned resid norm 8.397302896056e-04 true resid norm 8.397302896054e-04 ||r(i)||/||b|| 5.080093234850e-04 9 KSP unpreconditioned resid norm 6.264979217511e-04 true resid norm 6.264979217521e-04 ||r(i)||/||b|| 3.790107244358e-04 10 KSP unpreconditioned resid norm 4.502188878110e-04 true resid norm 4.502188878118e-04 ||r(i)||/||b|| 2.723676821577e-04 11 KSP unpreconditioned resid norm 3.486290906189e-04 true resid norm 3.486290906193e-04 ||r(i)||/||b|| 2.109091819898e-04 12 KSP unpreconditioned resid norm 2.892294177841e-04 true resid norm 2.892294177842e-04 ||r(i)||/||b|| 1.749743253034e-04 13 KSP unpreconditioned resid norm 2.455836801619e-04 true resid norm 2.455836801616e-04 ||r(i)||/||b|| 1.485700834687e-04 14 KSP unpreconditioned resid norm 2.080058928662e-04 true resid norm 2.080058928656e-04 ||r(i)||/||b|| 1.258367528522e-04 15 KSP unpreconditioned resid norm 1.769405745688e-04 true resid norm 1.769405745682e-04 ||r(i)||/||b|| 1.070432526921e-04 16 KSP unpreconditioned resid norm 1.532177881785e-04 true resid norm 1.532177881780e-04 ||r(i)||/||b|| 9.269174386308e-05 17 KSP unpreconditioned resid norm 1.213202871017e-04 true resid norm 1.213202871011e-04 ||r(i)||/||b|| 7.339480037595e-05 18 KSP unpreconditioned resid norm 1.008586917109e-04 true resid norm 1.008586917102e-04 ||r(i)||/||b|| 6.101620529536e-05 19 KSP unpreconditioned resid norm 7.536678882281e-05 true resid norm 7.536678882237e-05 ||r(i)||/||b|| 4.559443892500e-05 20 KSP unpreconditioned resid norm 6.165856591295e-05 true resid norm 6.165856591247e-05 ||r(i)||/||b|| 3.730141301795e-05 21 KSP unpreconditioned resid norm 4.781675773685e-05 true resid norm 4.781675773624e-05 ||r(i)||/||b|| 2.892757239976e-05 22 KSP unpreconditioned resid norm 4.111930662611e-05 true resid norm 4.111930662552e-05 ||r(i)||/||b|| 2.487583382376e-05 23 KSP unpreconditioned resid norm 3.231084932122e-05 true resid norm 3.231084932060e-05 ||r(i)||/||b|| 1.954700563712e-05 24 KSP unpreconditioned resid norm 2.629119845771e-05 true resid norm 2.629119845736e-05 ||r(i)||/||b|| 1.590531401244e-05 25 KSP unpreconditioned resid norm 2.062134326649e-05 true resid norm 2.062134326590e-05 ||r(i)||/||b|| 1.247523731314e-05 26 KSP unpreconditioned resid norm 1.598609489324e-05 true resid norm 1.598609489274e-05 ||r(i)||/||b|| 9.671063854853e-06 27 KSP unpreconditioned resid norm 1.293327139531e-05 true resid norm 1.293327139479e-05 ||r(i)||/||b|| 7.824205620597e-06 28 KSP unpreconditioned resid norm 1.035949175128e-05 true resid norm 1.035949175088e-05 ||r(i)||/||b|| 6.267153229026e-06 29 KSP unpreconditioned resid norm 8.880493935672e-06 true resid norm 8.880493935523e-06 ||r(i)||/||b|| 5.372407988900e-06 30 KSP unpreconditioned resid norm 7.137871934412e-06 true resid norm 7.137871934412e-06 ||r(i)||/||b|| 4.318178750259e-06 31 KSP unpreconditioned resid norm 6.707215155334e-06 true resid norm 6.707215155334e-06 ||r(i)||/||b|| 4.057645503213e-06 32 KSP unpreconditioned resid norm 5.556103817925e-06 true resid norm 5.556103817933e-06 ||r(i)||/||b|| 3.361260843748e-06 33 KSP unpreconditioned resid norm 4.988355482067e-06 true resid norm 4.988355482078e-06 ||r(i)||/||b|| 3.017791694693e-06 34 KSP unpreconditioned resid norm 4.087915820136e-06 true resid norm 4.087915820152e-06 ||r(i)||/||b|| 2.473055189226e-06 35 KSP unpreconditioned resid norm 3.481285629031e-06 true resid norm 3.481285629045e-06 ||r(i)||/||b|| 2.106063791149e-06 36 KSP unpreconditioned resid norm 2.775692176766e-06 true resid norm 2.775692176776e-06 ||r(i)||/||b|| 1.679202861182e-06 37 KSP unpreconditioned resid norm 2.326242471264e-06 true resid norm 2.326242471274e-06 ||r(i)||/||b|| 1.407300509130e-06 38 KSP unpreconditioned resid norm 1.903169493624e-06 true resid norm 1.903169493629e-06 ||r(i)||/||b|| 1.151355213577e-06 39 KSP unpreconditioned resid norm 1.647136962470e-06 true resid norm 1.647136962479e-06 ||r(i)||/||b|| 9.964639174670e-07 40 KSP unpreconditioned resid norm 1.341776814255e-06 true resid norm 1.341776814265e-06 ||r(i)||/||b|| 8.117310285457e-07 41 KSP unpreconditioned resid norm 1.113514152334e-06 true resid norm 1.113514152345e-06 ||r(i)||/||b|| 6.736395938384e-07 42 KSP unpreconditioned resid norm 8.915555405500e-07 true resid norm 8.915555405616e-07 ||r(i)||/||b|| 5.393619030020e-07 43 KSP unpreconditioned resid norm 7.237736566449e-07 true resid norm 7.237736566594e-07 ||r(i)||/||b|| 4.378593582096e-07 44 KSP unpreconditioned resid norm 5.921887748494e-07 true resid norm 5.921887748552e-07 ||r(i)||/||b|| 3.582548142106e-07 45 KSP unpreconditioned resid norm 4.832497255683e-07 true resid norm 4.832497255690e-07 ||r(i)||/||b|| 2.923502572189e-07 46 KSP unpreconditioned resid norm 4.072281865093e-07 true resid norm 4.072281865119e-07 ||r(i)||/||b|| 2.463597158453e-07 47 KSP unpreconditioned resid norm 3.355562061436e-07 true resid norm 3.355562061460e-07 ||r(i)||/||b|| 2.030005150291e-07 48 KSP unpreconditioned resid norm 2.849049830237e-07 true resid norm 2.849049830229e-07 ||r(i)||/||b|| 1.723581839009e-07 49 KSP unpreconditioned resid norm 2.266132667399e-07 true resid norm 2.266132667368e-07 ||r(i)||/||b|| 1.370936046403e-07 50 KSP unpreconditioned resid norm 1.739841395163e-07 true resid norm 1.739841395054e-07 ||r(i)||/||b|| 1.052547063043e-07 51 KSP unpreconditioned resid norm 1.268356186763e-07 true resid norm 1.268356186611e-07 ||r(i)||/||b|| 7.673139533900e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 1.26836e-07 Rank#44, time step = 3, continuity = 1.368123e-02 @ (5, 5, 4) Rank#24, time step = 3, continuity = 3.885754e-04 @ (41, 4, 4) Rank#10, time step = 3, continuity = 4.105729e-03 @ (4, 5, 35) Rank#3, time step = 3, continuity = 5.609826e-02 @ (4, 51, 35) Rank#45, time step = 3, continuity = 1.929346e-03 @ (4, 50, 34) Rank#27, time step = 3, continuity = 3.546324e+00 @ (4, 11, 17) Rank#11, time step = 3, continuity = 5.897940e-02 @ (4, 4, 35) Rank#2, time step = 3, continuity = 4.119999e-03 @ (4, 51, 35) Rank#0, time step = 3, continuity = 1.960288e-08 @ (19, 51, 7) Rank#46, time step = 3, continuity = 1.202898e-03 @ (4, 50, 34) Rank#8, time step = 3, continuity = 4.222265e-05 @ (41, 4, 35) Rank#4, time step = 3, continuity = 9.702817e-03 @ (4, 51, 35) Rank#5, time step = 3, continuity = 2.042632e-03 @ (4, 4, 4) Rank#6, time step = 3, continuity = 1.251370e-03 @ (4, 4, 4) Rank#30, time step = 3, continuity = 1.158913e-03 @ (4, 50, 5) Rank#14, time step = 3, continuity = 1.228256e-03 @ (4, 50, 4) Rank#7, time step = 3, continuity = 1.121091e-03 @ (4, 4, 4) Rank#15, time step = 3, continuity = 1.112731e-03 @ (4, 50, 4) Rank#16, time step = 3, continuity = 3.469996e-04 @ (41, 51, 4) Rank#17, time step = 3, continuity = 5.357466e-03 @ (41, 51, 4) Rank#18, time step = 3, continuity = 1.080270e-01 @ (39, 51, 4) Rank#25, time step = 3, continuity = 5.597218e-03 @ (41, 4, 4) Rank#13, time step = 3, continuity = 1.989096e-03 @ (4, 50, 4) Rank#12, time step = 3, continuity = 1.013889e-02 @ (4, 4, 35) Rank#26, time step = 3, continuity = 7.539334e-01 @ (39, 4, 17) Rank#23, time step = 3, continuity = 1.112361e-03 @ (4, 4, 5) Rank#20, time step = 3, continuity = 2.731360e-02 @ (4, 51, 20) Rank#22, time step = 3, continuity = 1.184401e-03 @ (4, 4, 5) Rank#21, time step = 3, continuity = 1.597810e-03 @ (4, 4, 5) Rank#39, time step = 3, continuity = 1.112194e-03 @ (4, 5, 34) Rank#36, time step = 3, continuity = 1.367959e-02 @ (5, 51, 4) Rank#37, time step = 3, continuity = 1.986552e-03 @ (4, 4, 34) Rank#40, time step = 3, continuity = 2.850005e-04 @ (41, 5, 4) Rank#28, time step = 3, continuity = 2.828638e-02 @ (4, 4, 20) Rank#38, time step = 3, continuity = 1.226979e-03 @ (4, 4, 34) Rank#41, time step = 3, continuity = 5.161523e-03 @ (41, 5, 4) Rank#29, time step = 3, continuity = 1.517670e-03 @ (4, 50, 5) Rank#42, time step = 3, continuity = 1.294414e-01 @ (40, 5, 4) Rank#43, time step = 3, continuity = 1.224322e-01 @ (19, 5, 4) Rank#32, time step = 3, continuity = 2.852451e-04 @ (41, 51, 4) Rank#31, time step = 3, continuity = 1.103723e-03 @ (4, 50, 5) Rank#33, time step = 3, continuity = 5.164929e-03 @ (41, 51, 4) Rank#34, time step = 3, continuity = 1.287048e-01 @ (40, 51, 4) Rank#9, time step = 3, continuity = 2.683230e-04 @ (4, 50, 4) Rank#1, time step = 3, continuity = 2.941918e-04 @ (4, 4, 4) Rank#19, time step = 3, continuity = 3.689953e+00 @ (4, 51, 13) Rank#47, time step = 3, continuity = 1.103518e-03 @ (4, 50, 34) Rank#35, time step = 3, continuity = 1.214269e-01 @ (19, 51, 4) Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 0 KSP unpreconditioned resid norm 1.800295456670e+01 true resid norm 1.800295456670e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.295938894634e+00 true resid norm 1.295938894634e+00 ||r(i)||/||b|| 7.198478948734e-02 2 KSP unpreconditioned resid norm 3.597174991333e-01 true resid norm 3.597174991333e-01 ||r(i)||/||b|| 1.998102576999e-02 3 KSP unpreconditioned resid norm 1.228849356110e-01 true resid norm 1.228849356110e-01 ||r(i)||/||b|| 6.825820459396e-03 4 KSP unpreconditioned resid norm 5.526977676103e-02 true resid norm 5.526977676104e-02 ||r(i)||/||b|| 3.070039229186e-03 5 KSP unpreconditioned resid norm 3.618819388233e-02 true resid norm 3.618819388233e-02 ||r(i)||/||b|| 2.010125268508e-03 6 KSP unpreconditioned resid norm 2.416631959864e-02 true resid norm 2.416631959864e-02 ||r(i)||/||b|| 1.342352973736e-03 7 KSP unpreconditioned resid norm 1.499930293446e-02 true resid norm 1.499930293447e-02 ||r(i)||/||b|| 8.331578507791e-04 8 KSP unpreconditioned resid norm 1.055050367432e-02 true resid norm 1.055050367432e-02 ||r(i)||/||b|| 5.860428984156e-04 9 KSP unpreconditioned resid norm 7.822204060520e-03 true resid norm 7.822204060528e-03 ||r(i)||/||b|| 4.344955730208e-04 10 KSP unpreconditioned resid norm 5.732827542959e-03 true resid norm 5.732827542962e-03 ||r(i)||/||b|| 3.184381497893e-04 11 KSP unpreconditioned resid norm 4.349201692653e-03 true resid norm 4.349201692655e-03 ||r(i)||/||b|| 2.415826622536e-04 12 KSP unpreconditioned resid norm 3.507456574416e-03 true resid norm 3.507456574419e-03 ||r(i)||/||b|| 1.948267192156e-04 13 KSP unpreconditioned resid norm 2.714359769664e-03 true resid norm 2.714359769663e-03 ||r(i)||/||b|| 1.507730167071e-04 14 KSP unpreconditioned resid norm 2.159613131461e-03 true resid norm 2.159613131459e-03 ||r(i)||/||b|| 1.199588169518e-04 15 KSP unpreconditioned resid norm 1.704473557682e-03 true resid norm 1.704473557679e-03 ||r(i)||/||b|| 9.467743482680e-05 16 KSP unpreconditioned resid norm 1.390545859097e-03 true resid norm 1.390545859096e-03 ||r(i)||/||b|| 7.723986937498e-05 17 KSP unpreconditioned resid norm 1.077180358743e-03 true resid norm 1.077180358741e-03 ||r(i)||/||b|| 5.983353203220e-05 18 KSP unpreconditioned resid norm 9.048346152026e-04 true resid norm 9.048346152024e-04 ||r(i)||/||b|| 5.026033987087e-05 19 KSP unpreconditioned resid norm 7.300819022225e-04 true resid norm 7.300819022251e-04 ||r(i)||/||b|| 4.055344913081e-05 20 KSP unpreconditioned resid norm 6.034455779373e-04 true resid norm 6.034455779385e-04 ||r(i)||/||b|| 3.351925239286e-05 21 KSP unpreconditioned resid norm 5.027864502361e-04 true resid norm 5.027864502383e-04 ||r(i)||/||b|| 2.792799639500e-05 22 KSP unpreconditioned resid norm 4.185448641713e-04 true resid norm 4.185448641737e-04 ||r(i)||/||b|| 2.324867635604e-05 23 KSP unpreconditioned resid norm 3.383127450993e-04 true resid norm 3.383127451025e-04 ||r(i)||/||b|| 1.879206792691e-05 24 KSP unpreconditioned resid norm 2.658271933588e-04 true resid norm 2.658271933632e-04 ||r(i)||/||b|| 1.476575371994e-05 25 KSP unpreconditioned resid norm 2.086016977258e-04 true resid norm 2.086016977289e-04 ||r(i)||/||b|| 1.158708127358e-05 26 KSP unpreconditioned resid norm 1.553878432255e-04 true resid norm 1.553878432301e-04 ||r(i)||/||b|| 8.631241202902e-06 27 KSP unpreconditioned resid norm 1.207652470263e-04 true resid norm 1.207652470298e-04 ||r(i)||/||b|| 6.708079308999e-06 28 KSP unpreconditioned resid norm 9.478447325293e-05 true resid norm 9.478447325797e-05 ||r(i)||/||b|| 5.264939868996e-06 29 KSP unpreconditioned resid norm 7.309077489139e-05 true resid norm 7.309077489702e-05 ||r(i)||/||b|| 4.059932197585e-06 30 KSP unpreconditioned resid norm 5.488099653620e-05 true resid norm 5.488099653620e-05 ||r(i)||/||b|| 3.048443872524e-06 31 KSP unpreconditioned resid norm 4.691702649096e-05 true resid norm 4.691702649093e-05 ||r(i)||/||b|| 2.606073704019e-06 32 KSP unpreconditioned resid norm 3.921717594840e-05 true resid norm 3.921717594834e-05 ||r(i)||/||b|| 2.178374433098e-06 33 KSP unpreconditioned resid norm 3.340909038588e-05 true resid norm 3.340909038575e-05 ||r(i)||/||b|| 1.855755968387e-06 34 KSP unpreconditioned resid norm 2.772109424893e-05 true resid norm 2.772109424866e-05 ||r(i)||/||b|| 1.539808043505e-06 35 KSP unpreconditioned resid norm 2.185592027217e-05 true resid norm 2.185592027141e-05 ||r(i)||/||b|| 1.214018520706e-06 36 KSP unpreconditioned resid norm 1.903747666486e-05 true resid norm 1.903747666396e-05 ||r(i)||/||b|| 1.057464017555e-06 37 KSP unpreconditioned resid norm 1.624600448157e-05 true resid norm 1.624600448058e-05 ||r(i)||/||b|| 9.024076809383e-07 38 KSP unpreconditioned resid norm 1.420940242615e-05 true resid norm 1.420940242508e-05 ||r(i)||/||b|| 7.892816910929e-07 39 KSP unpreconditioned resid norm 1.203584566250e-05 true resid norm 1.203584566145e-05 ||r(i)||/||b|| 6.685483550411e-07 40 KSP unpreconditioned resid norm 1.020862316387e-05 true resid norm 1.020862316287e-05 ||r(i)||/||b|| 5.670526537766e-07 41 KSP unpreconditioned resid norm 8.241595972505e-06 true resid norm 8.241595971594e-06 ||r(i)||/||b|| 4.577912998146e-07 42 KSP unpreconditioned resid norm 6.689625267539e-06 true resid norm 6.689625266635e-06 ||r(i)||/||b|| 3.715848552441e-07 43 KSP unpreconditioned resid norm 5.248454162379e-06 true resid norm 5.248454161620e-06 ||r(i)||/||b|| 2.915329337845e-07 44 KSP unpreconditioned resid norm 3.931049414091e-06 true resid norm 3.931049413544e-06 ||r(i)||/||b|| 2.183557925994e-07 45 KSP unpreconditioned resid norm 3.002868065889e-06 true resid norm 3.002868065505e-06 ||r(i)||/||b|| 1.667986248801e-07 46 KSP unpreconditioned resid norm 2.243200121528e-06 true resid norm 2.243200121288e-06 ||r(i)||/||b|| 1.246017765016e-07 47 KSP unpreconditioned resid norm 1.693164287854e-06 true resid norm 1.693164287781e-06 ||r(i)||/||b|| 9.404924516737e-08 Linear solve converged due to CONVERGED_RTOL iterations 47 Residual norm 1.69316e-06 0 KSP unpreconditioned resid norm 1.412673063450e+00 true resid norm 1.412673063450e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 8.531572199357e-02 true resid norm 8.531572199358e-02 ||r(i)||/||b|| 6.039311161296e-02 2 KSP unpreconditioned resid norm 2.354905984408e-02 true resid norm 2.354905984408e-02 ||r(i)||/||b|| 1.666985833691e-02 3 KSP unpreconditioned resid norm 8.866561466319e-03 true resid norm 8.866561466321e-03 ||r(i)||/||b|| 6.276442650265e-03 4 KSP unpreconditioned resid norm 4.007731044735e-03 true resid norm 4.007731044736e-03 ||r(i)||/||b|| 2.836984117859e-03 5 KSP unpreconditioned resid norm 2.562270094262e-03 true resid norm 2.562270094262e-03 ||r(i)||/||b|| 1.813774298211e-03 6 KSP unpreconditioned resid norm 1.648374750802e-03 true resid norm 1.648374750802e-03 ||r(i)||/||b|| 1.166848008538e-03 7 KSP unpreconditioned resid norm 1.012675849200e-03 true resid norm 1.012675849200e-03 ||r(i)||/||b|| 7.168508237332e-04 8 KSP unpreconditioned resid norm 6.975172897121e-04 true resid norm 6.975172897118e-04 ||r(i)||/||b|| 4.937570537433e-04 9 KSP unpreconditioned resid norm 5.194036995358e-04 true resid norm 5.194036995367e-04 ||r(i)||/||b|| 3.676743848064e-04 10 KSP unpreconditioned resid norm 3.696092053889e-04 true resid norm 3.696092053896e-04 ||r(i)||/||b|| 2.616381772629e-04 11 KSP unpreconditioned resid norm 2.850523803892e-04 true resid norm 2.850523803895e-04 ||r(i)||/||b|| 2.017822720377e-04 12 KSP unpreconditioned resid norm 2.350800134299e-04 true resid norm 2.350800134300e-04 ||r(i)||/||b|| 1.664079393259e-04 13 KSP unpreconditioned resid norm 2.005287837193e-04 true resid norm 2.005287837191e-04 ||r(i)||/||b|| 1.419498884117e-04 14 KSP unpreconditioned resid norm 1.693423716447e-04 true resid norm 1.693423716441e-04 ||r(i)||/||b|| 1.198737174407e-04 15 KSP unpreconditioned resid norm 1.450875061234e-04 true resid norm 1.450875061229e-04 ||r(i)||/||b|| 1.027042348840e-04 16 KSP unpreconditioned resid norm 1.255170696894e-04 true resid norm 1.255170696889e-04 ||r(i)||/||b|| 8.885075601452e-05 17 KSP unpreconditioned resid norm 1.001150732534e-04 true resid norm 1.001150732528e-04 ||r(i)||/||b|| 7.086924486848e-05 18 KSP unpreconditioned resid norm 8.324107526331e-05 true resid norm 8.324107526263e-05 ||r(i)||/||b|| 5.892451510282e-05 19 KSP unpreconditioned resid norm 6.233954903979e-05 true resid norm 6.233954903940e-05 ||r(i)||/||b|| 4.412878722776e-05 20 KSP unpreconditioned resid norm 5.068242281259e-05 true resid norm 5.068242281210e-05 ||r(i)||/||b|| 3.587696553675e-05 21 KSP unpreconditioned resid norm 3.925070811210e-05 true resid norm 3.925070811142e-05 ||r(i)||/||b|| 2.778470767720e-05 22 KSP unpreconditioned resid norm 3.353801271807e-05 true resid norm 3.353801271731e-05 ||r(i)||/||b|| 2.374081702626e-05 23 KSP unpreconditioned resid norm 2.647759351203e-05 true resid norm 2.647759351127e-05 ||r(i)||/||b|| 1.874290251319e-05 24 KSP unpreconditioned resid norm 2.135260655706e-05 true resid norm 2.135260655663e-05 ||r(i)||/||b|| 1.511503766093e-05 25 KSP unpreconditioned resid norm 1.687070927407e-05 true resid norm 1.687070927336e-05 ||r(i)||/||b|| 1.194240175582e-05 26 KSP unpreconditioned resid norm 1.293694698390e-05 true resid norm 1.293694698317e-05 ||r(i)||/||b|| 9.157778482429e-06 27 KSP unpreconditioned resid norm 1.049854679096e-05 true resid norm 1.049854679022e-05 ||r(i)||/||b|| 7.431688946185e-06 28 KSP unpreconditioned resid norm 8.323105069998e-06 true resid norm 8.323105069154e-06 ||r(i)||/||b|| 5.891741893078e-06 29 KSP unpreconditioned resid norm 7.149131008665e-06 true resid norm 7.149131007929e-06 ||r(i)||/||b|| 5.060711634486e-06 30 KSP unpreconditioned resid norm 5.765627614985e-06 true resid norm 5.765627614985e-06 ||r(i)||/||b|| 4.081360198730e-06 31 KSP unpreconditioned resid norm 5.420427196404e-06 true resid norm 5.420427196403e-06 ||r(i)||/||b|| 3.837000461497e-06 32 KSP unpreconditioned resid norm 4.532576209396e-06 true resid norm 4.532576209394e-06 ||r(i)||/||b|| 3.208510395408e-06 33 KSP unpreconditioned resid norm 4.064657789569e-06 true resid norm 4.064657789567e-06 ||r(i)||/||b|| 2.877281300770e-06 34 KSP unpreconditioned resid norm 3.349298785897e-06 true resid norm 3.349298785894e-06 ||r(i)||/||b|| 2.370894492540e-06 35 KSP unpreconditioned resid norm 2.848140811778e-06 true resid norm 2.848140811775e-06 ||r(i)||/||b|| 2.016135852990e-06 36 KSP unpreconditioned resid norm 2.272096193713e-06 true resid norm 2.272096193716e-06 ||r(i)||/||b|| 1.608366615391e-06 37 KSP unpreconditioned resid norm 1.902087742756e-06 true resid norm 1.902087742762e-06 ||r(i)||/||b|| 1.346445821028e-06 38 KSP unpreconditioned resid norm 1.556881030367e-06 true resid norm 1.556881030374e-06 ||r(i)||/||b|| 1.102081628548e-06 39 KSP unpreconditioned resid norm 1.348642308105e-06 true resid norm 1.348642308109e-06 ||r(i)||/||b|| 9.546740452567e-07 40 KSP unpreconditioned resid norm 1.100324584644e-06 true resid norm 1.100324584650e-06 ||r(i)||/||b|| 7.788954239433e-07 41 KSP unpreconditioned resid norm 9.053973900964e-07 true resid norm 9.053973900970e-07 ||r(i)||/||b|| 6.409107765427e-07 42 KSP unpreconditioned resid norm 7.230581243700e-07 true resid norm 7.230581243733e-07 ||r(i)||/||b|| 5.118368453966e-07 43 KSP unpreconditioned resid norm 5.825287157867e-07 true resid norm 5.825287157901e-07 ||r(i)||/||b|| 4.123591868930e-07 44 KSP unpreconditioned resid norm 4.762644417313e-07 true resid norm 4.762644417360e-07 ||r(i)||/||b|| 3.371370588555e-07 45 KSP unpreconditioned resid norm 3.865941721477e-07 true resid norm 3.865941721540e-07 ||r(i)||/||b|| 2.736614593682e-07 46 KSP unpreconditioned resid norm 3.263216396424e-07 true resid norm 3.263216396447e-07 ||r(i)||/||b|| 2.309958674003e-07 47 KSP unpreconditioned resid norm 2.681743797359e-07 true resid norm 2.681743797427e-07 ||r(i)||/||b|| 1.898347088800e-07 48 KSP unpreconditioned resid norm 2.281833238064e-07 true resid norm 2.281833238130e-07 ||r(i)||/||b|| 1.615259253657e-07 49 KSP unpreconditioned resid norm 1.815350740360e-07 true resid norm 1.815350740409e-07 ||r(i)||/||b|| 1.285046616501e-07 50 KSP unpreconditioned resid norm 1.392510493034e-07 true resid norm 1.392510493056e-07 ||r(i)||/||b|| 9.857273626037e-08 Linear solve converged due to CONVERGED_RTOL iterations 50 Residual norm 1.39251e-07 Rank#47, time step = 4, continuity = 9.072241e-04 @ (4, 50, 34) Rank#32, time step = 4, continuity = 2.993468e-04 @ (41, 51, 4) Rank#34, time step = 4, continuity = 1.409515e-01 @ (40, 51, 4) Rank#0, time step = 4, continuity = 2.806779e-08 @ (41, 51, 35) Rank#9, time step = 4, continuity = 3.242195e-04 @ (4, 50, 4) Rank#1, time step = 4, continuity = 3.559030e-04 @ (4, 4, 4) Rank#10, time step = 4, continuity = 4.821606e-03 @ (4, 5, 35) Rank#27, time step = 4, continuity = 3.181760e+00 @ (4, 8, 14) Rank#36, time step = 4, continuity = 1.548400e-02 @ (5, 51, 4) Rank#37, time step = 4, continuity = 1.847557e-03 @ (4, 4, 34) Rank#8, time step = 4, continuity = 4.587898e-05 @ (41, 4, 35) Rank#44, time step = 4, continuity = 1.552645e-02 @ (5, 5, 4) Rank#16, time step = 4, continuity = 3.709613e-04 @ (41, 51, 4) Rank#4, time step = 4, continuity = 1.173989e-02 @ (4, 51, 35) Rank#45, time step = 4, continuity = 1.782619e-03 @ (4, 50, 34) Rank#18, time step = 4, continuity = 1.182176e-01 @ (39, 51, 4) Rank#5, time step = 4, continuity = 1.908359e-03 @ (4, 4, 4) Rank#46, time step = 4, continuity = 9.872820e-04 @ (4, 50, 34) Rank#6, time step = 4, continuity = 1.039245e-03 @ (4, 4, 4) Rank#12, time step = 4, continuity = 1.240915e-02 @ (4, 4, 35) Rank#25, time step = 4, continuity = 6.008582e-03 @ (41, 4, 4) Rank#13, time step = 4, continuity = 1.847262e-03 @ (4, 50, 4) Rank#14, time step = 4, continuity = 1.013285e-03 @ (4, 50, 4) Rank#24, time step = 4, continuity = 4.176380e-04 @ (41, 4, 4) Rank#40, time step = 4, continuity = 2.993544e-04 @ (41, 5, 4) Rank#42, time step = 4, continuity = 1.417756e-01 @ (40, 5, 4) Rank#28, time step = 4, continuity = 3.335993e-02 @ (4, 4, 20) Rank#43, time step = 4, continuity = 1.360899e-01 @ (19, 5, 4) Rank#29, time step = 4, continuity = 1.310806e-03 @ (4, 50, 5) Rank#33, time step = 4, continuity = 5.532869e-03 @ (41, 51, 4) Rank#23, time step = 4, continuity = 9.159324e-04 @ (4, 4, 5) Rank#2, time step = 4, continuity = 4.837666e-03 @ (4, 51, 35) Rank#20, time step = 4, continuity = 3.225439e-02 @ (4, 51, 20) Rank#21, time step = 4, continuity = 1.402715e-03 @ (4, 4, 5) Rank#22, time step = 4, continuity = 9.657753e-04 @ (4, 4, 5) Rank#26, time step = 4, continuity = 6.323174e-01 @ (39, 4, 17) Rank#19, time step = 4, continuity = 3.372144e+00 @ (4, 51, 13) Rank#30, time step = 4, continuity = 9.366603e-04 @ (4, 50, 5) Rank#41, time step = 4, continuity = 5.527059e-03 @ (41, 5, 4) Rank#38, time step = 4, continuity = 1.013971e-03 @ (4, 4, 34) Rank#11, time step = 4, continuity = 6.808021e-02 @ (4, 4, 35) Rank#31, time step = 4, continuity = 9.066095e-04 @ (4, 50, 5) Rank#17, time step = 4, continuity = 5.747098e-03 @ (41, 51, 4) Rank#39, time step = 4, continuity = 9.165216e-04 @ (4, 5, 34) Rank#3, time step = 4, continuity = 6.498223e-02 @ (4, 51, 35) Rank#7, time step = 4, continuity = 9.252253e-04 @ (4, 4, 4) Rank#15, time step = 4, continuity = 9.161437e-04 @ (4, 50, 4) Rank#35, time step = 4, continuity = 1.349255e-01 @ (19, 51, 4) ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex45 on a arch-linux2-c-debug named compute-8-8.local with 48 processors, by zlwei Fri Oct 5 14:00:22 2012 Using Petsc Development HG revision: 98bf11863c3be31b7c2af504314a500bc64d88c9 HG Date: Wed Aug 29 13:51:08 2012 -0500 Max Max/Min Avg Total Time (sec): 1.246e+02 1.00015 1.246e+02 Objects: 5.499e+03 1.00000 5.499e+03 Flops: 8.761e+09 1.03903 8.570e+09 4.114e+11 Flops/sec: 7.034e+07 1.03901 6.881e+07 3.303e+09 MPI Messages: 1.089e+05 2.25599 7.804e+04 3.746e+06 MPI Message Lengths: 4.049e+08 1.75931 4.110e+03 1.540e+10 MPI Reductions: 2.272e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.0932e+01 32.9% 4.6734e+08 0.1% 1.037e+04 0.3% 1.167e+03 28.4% 3.200e+01 0.1% 1: DMMG Setup: 3.5440e+00 2.8% 0.0000e+00 0.0% 1.528e+03 0.0% 6.567e-01 0.0% 3.500e+01 0.2% 2: Pressure RHS Setup: 1.0369e+01 8.3% 1.4382e+10 3.5% 3.013e+05 8.0% 2.590e+02 6.3% 3.635e+03 16.0% 3: Pressure Solve: 2.8890e+01 23.2% 1.8356e+11 44.6% 1.513e+06 40.4% 1.171e+03 28.5% 7.453e+03 32.8% 4: Corrector RHS Setup: 9.9616e+00 8.0% 1.4382e+10 3.5% 3.013e+05 8.0% 2.590e+02 6.3% 3.635e+03 16.0% 5: Corrector Solve: 3.0856e+01 24.8% 1.9859e+11 48.3% 1.619e+06 43.2% 1.253e+03 30.5% 7.929e+03 34.9% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecNorm 10 1.0 5.0277e-02 2.3 1.17e+06 1.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 12 0 0 31 1100 VecSet 1 1.0 2.1939e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 10 1.0 2.4841e-03 1.6 1.17e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 12 0 0 0 22260 VecScatterBegin 11 1.0 5.4202e-03 3.0 0.00e+00 0.0 2.5e+03 1.2e+04 0.0e+00 0 0 0 0 0 0 0 24 1 0 0 VecScatterEnd 11 1.0 2.4676e-0213.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 10 1.0 4.8111e-02 1.7 7.56e+06 1.0 2.0e+03 1.2e+04 0.0e+00 0 0 0 0 0 0 76 19 1 0 7415 --- Event Stage 1: DMMG Setup --- Event Stage 2: Pressure RHS Setup KSPGMRESOrthog 150 1.0 4.9527e-01 1.7 7.04e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 1 0 0 1 4 23 0 0 4 6736 KSPSetUp 40 1.0 2.1560e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+02 0 0 0 0 1 2 0 0 0 5 0 VecMDot 150 1.0 4.3920e-01 1.9 3.52e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 3 12 0 0 4 3798 VecNorm 165 1.0 3.3326e-01 2.4 7.04e+06 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 2 0 0 5 1001 VecScale 165 1.0 1.0014e-02 2.8 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 16657 VecCopy 15 1.0 1.6418e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 235 1.0 7.2906e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 15 1.0 1.8489e-03 2.1 6.40e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16402 VecMAXPY 165 1.0 9.6994e-02 2.0 4.16e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 14 0 0 0 20323 VecAssemblyBegin 215 1.0 6.1984e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.3e+02 0 0 0 0 3 5 0 0 0 17 0 VecAssemblyEnd 215 1.0 4.8971e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 165 1.0 3.1083e-02 1.8 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 5366 VecScatterBegin 375 1.0 6.6681e-02 3.3 0.00e+00 0.0 1.7e+05 3.4e+03 0.0e+00 0 0 5 4 0 0 0 56 60 0 0 VecScatterEnd 375 1.0 5.2347e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0 VecSetRandom 15 1.0 1.0735e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 165 1.0 3.4388e-01 2.3 1.06e+07 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 3 0 0 5 1455 MatMult 150 1.0 3.7738e-01 2.0 4.51e+07 1.0 5.7e+04 2.8e+03 0.0e+00 0 1 2 1 0 3 15 19 16 0 5631 MatConvert 15 1.0 6.1136e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatScale 15 1.0 1.3903e-01 4.2 4.83e+06 1.0 5.7e+03 2.8e+03 0.0e+00 0 0 0 0 0 1 2 2 2 0 1638 MatAssemblyBegin 195 1.0 1.5021e+00 1.4 0.00e+00 0.0 1.3e+04 6.0e+02 2.2e+02 1 0 0 0 1 13 0 4 1 6 0 MatAssemblyEnd 195 1.0 1.0008e+00 1.3 0.00e+00 0.0 5.2e+04 6.3e+02 6.0e+02 1 0 1 0 3 8 0 17 3 17 0 MatGetRow 639730 1.0 1.6904e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 MatCoarsen 15 1.0 6.1620e-01 1.0 0.00e+00 0.0 5.3e+04 3.7e+03 4.3e+02 0 0 1 1 2 6 0 18 20 12 0 MatPtAP 15 1.0 7.3923e-01 1.1 1.40e+07 1.1 4.8e+04 1.0e+03 4.0e+02 1 0 1 0 2 7 5 16 5 11 886 MatPtAPSymbolic 15 1.0 5.5060e-01 1.1 0.00e+00 0.0 4.4e+04 9.0e+02 3.8e+02 0 0 1 0 2 5 0 15 4 10 0 MatPtAPNumeric 15 1.0 2.1724e-01 1.3 1.40e+07 1.1 4.3e+03 2.2e+03 3.0e+01 0 0 0 0 0 2 5 1 1 1 3014 MatTrnMatMult 15 1.0 3.4586e+00 1.0 1.51e+08 1.1 3.9e+04 7.8e+03 4.4e+02 3 2 1 2 2 33 49 13 31 12 2035 MatGetLocalMat 45 1.0 1.1697e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 15 1.0 1.0516e-01 2.3 0.00e+00 0.0 1.7e+04 1.8e+03 3.0e+01 0 0 0 0 0 1 0 6 3 1 0 MatGetSymTrans 30 1.0 5.6214e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.6096e+00 1.0 3.07e+08 1.0 2.9e+05 3.3e+03 3.2e+03 7 3 8 6 14 83100 97 98 89 1670 PCGAMGgraph_AGG 3 1.0 3.7921e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 120 PCGAMGcoarse_AGG 3 1.0 1.0288e+00 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 10 10 9 13 6 1368 PCGAMGProl_AGG 3 1.0 1.2415e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.9839e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 153 PCGAMGcoarse_AGG 3 1.0 8.6779e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1622 PCGAMGProl_AGG 3 1.0 1.2736e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7482e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 166 PCGAMGcoarse_AGG 3 1.0 8.9944e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1565 PCGAMGProl_AGG 3 1.0 1.2325e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.5020e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.8151e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 162 PCGAMGcoarse_AGG 3 1.0 8.8194e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1596 PCGAMGProl_AGG 3 1.0 1.3077e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7213e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 167 PCGAMGcoarse_AGG 3 1.0 8.9080e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1580 PCGAMGProl_AGG 3 1.0 1.3203e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 5.9605e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 3: Pressure Solve KSPGMRESOrthog 1928 1.0 7.4579e+00 1.6 8.71e+08 1.0 0.0e+00 0.0e+00 1.9e+03 5 10 0 0 8 20 22 0 0 26 5531 KSPSetUp 20 1.0 2.8610e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 2.8903e+01 1.0 3.92e+09 1.0 1.5e+06 2.9e+03 7.5e+03 23 45 40 28 33 100100100100100 6351 VecMDot 1928 1.0 6.7700e+00 1.7 4.36e+08 1.0 0.0e+00 0.0e+00 1.9e+03 4 5 0 0 8 17 11 0 0 26 3046 VecNorm 3876 1.0 9.7277e+00 1.8 1.82e+08 1.0 0.0e+00 0.0e+00 3.9e+03 6 2 0 0 17 24 5 0 0 52 887 VecScale 3625 1.0 2.0565e-01 2.6 7.66e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 17592 VecCopy 1938 1.0 2.5364e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 7958 1.0 4.1689e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 2656 1.0 2.5585e-01 1.5 1.21e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 3 0 0 0 22458 VecAYPX 969 1.0 1.5139e-01 1.5 2.98e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 9321 VecWAXPY 5 1.0 2.1451e-03 1.6 2.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6445 VecMAXPY 3861 1.0 2.1212e+00 1.6 8.82e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 10 0 0 0 6 23 0 0 0 19683 VecAssemblyBegin 5 1.0 7.7687e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 1.7405e-05 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 5071 1.0 6.4226e-01 2.6 0.00e+00 0.0 1.5e+06 2.9e+03 0.0e+00 0 0 40 28 0 1 0100100 0 0 VecScatterEnd 5071 1.0 4.6577e+00 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 10 0 0 0 0 0 VecNormalize 3374 1.0 7.3255e+00 1.7 1.86e+08 1.0 0.0e+00 0.0e+00 3.4e+03 4 2 0 0 15 18 5 0 0 45 1197 MatMult 3625 1.0 9.2039e+00 1.6 1.24e+09 1.0 1.2e+06 3.5e+03 0.0e+00 6 14 32 27 0 27 32 79 96 0 6359 MatMultAdd 723 1.0 1.2736e+00 3.1 3.08e+07 1.0 1.6e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1148 MatMultTranspose 723 1.0 9.4210e-01 1.9 3.08e+07 1.0 1.6e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1552 MatSolve 3374 1.2 3.9078e+00 1.6 8.80e+08 1.1 0.0e+00 0.0e+00 0.0e+00 3 10 0 0 0 11 22 0 0 0 10213 MatLUFactorSym 5 1.0 6.0294e-0367.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 20 1.0 9.0770e-02 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4485 MatILUFactorSym 15 1.0 5.6968e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 20 1.3 4.4918e-0458.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 20 1.3 6.2945e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 20 1.0 1.5481e-01 1.6 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2630 PCSetUpOnBlocks 1687 1.0 1.5688e-01 1.5 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2595 PCApply 241 1.0 1.9936e+01 1.0 2.30e+09 1.1 1.4e+06 2.3e+03 5.1e+03 16 26 38 21 23 68 58 94 73 69 5361 --- Event Stage 4: Corrector RHS Setup KSPGMRESOrthog 150 1.0 4.4386e-01 1.5 7.04e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 1 0 0 1 4 23 0 0 4 7516 KSPSetUp 40 1.0 1.8739e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+02 0 0 0 0 1 2 0 0 0 5 0 VecMDot 150 1.0 3.9025e-01 1.7 3.52e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 3 12 0 0 4 4274 VecNorm 165 1.0 3.0658e-01 2.2 7.04e+06 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 2 0 0 5 1088 VecScale 165 1.0 1.0942e-02 3.1 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 15243 VecCopy 15 1.0 3.6764e-03 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 235 1.0 7.4570e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 15 1.0 1.7416e-03 1.9 6.40e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 17413 VecMAXPY 165 1.0 1.0165e-01 2.1 4.16e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 14 0 0 0 19393 VecAssemblyBegin 215 1.0 5.6306e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.3e+02 0 0 0 0 3 5 0 0 0 17 0 VecAssemblyEnd 215 1.0 1.0600e-03 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 165 1.0 3.2980e-02 2.0 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 5057 VecScatterBegin 375 1.0 6.3563e-02 3.1 0.00e+00 0.0 1.7e+05 3.4e+03 0.0e+00 0 0 5 4 0 0 0 56 60 0 0 VecScatterEnd 375 1.0 5.1996e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0 VecSetRandom 15 1.0 9.5236e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 165 1.0 3.1596e-01 2.2 1.06e+07 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 3 0 0 5 1584 MatMult 150 1.0 3.3380e-01 1.8 4.51e+07 1.0 5.7e+04 2.8e+03 0.0e+00 0 1 2 1 0 3 15 19 16 0 6366 MatConvert 15 1.0 5.5341e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatScale 15 1.0 9.3830e-02 3.1 4.83e+06 1.0 5.7e+03 2.8e+03 0.0e+00 0 0 0 0 0 1 2 2 2 0 2426 MatAssemblyBegin 195 1.0 1.3034e+00 1.5 0.00e+00 0.0 1.3e+04 6.0e+02 2.2e+02 1 0 0 0 1 11 0 4 1 6 0 MatAssemblyEnd 195 1.0 9.7578e-01 1.2 0.00e+00 0.0 5.2e+04 6.3e+02 6.0e+02 1 0 1 0 3 9 0 17 3 17 0 MatGetRow 639730 1.0 1.8638e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 MatCoarsen 15 1.0 6.0700e-01 1.1 0.00e+00 0.0 5.3e+04 3.7e+03 4.3e+02 0 0 1 1 2 6 0 18 20 12 0 MatPtAP 15 1.0 7.4058e-01 1.1 1.40e+07 1.1 4.8e+04 1.0e+03 4.0e+02 1 0 1 0 2 7 5 16 5 11 884 MatPtAPSymbolic 15 1.0 5.6122e-01 1.1 0.00e+00 0.0 4.4e+04 9.0e+02 3.8e+02 0 0 1 0 2 5 0 15 4 10 0 MatPtAPNumeric 15 1.0 2.1232e-01 1.3 1.40e+07 1.1 4.3e+03 2.2e+03 3.0e+01 0 0 0 0 0 2 5 1 1 1 3084 MatTrnMatMult 15 1.0 3.2677e+00 1.0 1.51e+08 1.1 3.9e+04 7.8e+03 4.4e+02 3 2 1 2 2 33 49 13 31 12 2154 MatGetLocalMat 45 1.0 1.1687e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 15 1.0 1.0224e-01 2.1 0.00e+00 0.0 1.7e+04 1.8e+03 3.0e+01 0 0 0 0 0 1 0 6 3 1 0 MatGetSymTrans 30 1.0 5.1825e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.2378e+00 1.0 3.07e+08 1.0 2.9e+05 3.3e+03 3.2e+03 7 3 8 6 14 83100 97 98 89 1746 PCGAMGgraph_AGG 3 1.0 2.6251e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 173 PCGAMGcoarse_AGG 3 1.0 8.3389e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1688 PCGAMGProl_AGG 3 1.0 1.2837e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7516e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 165 PCGAMGcoarse_AGG 3 1.0 8.5538e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1646 PCGAMGProl_AGG 3 1.0 1.3015e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.3113e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.8946e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 157 PCGAMGcoarse_AGG 3 1.0 8.7704e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1605 PCGAMGProl_AGG 3 1.0 1.3723e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.1921e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7662e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 165 PCGAMGcoarse_AGG 3 1.0 9.1107e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1545 PCGAMGProl_AGG 3 1.0 1.2597e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7047e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 168 PCGAMGcoarse_AGG 3 1.0 8.7131e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1615 PCGAMGProl_AGG 3 1.0 1.3009e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 5: Corrector Solve KSPGMRESOrthog 2064 1.0 7.7906e+00 1.5 9.62e+08 1.0 0.0e+00 0.0e+00 2.1e+03 5 11 0 0 9 20 23 0 0 26 5847 KSPSetUp 20 1.0 3.0756e-05 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 3.0870e+01 1.0 4.24e+09 1.0 1.6e+06 2.9e+03 7.9e+03 25 48 43 30 35 100100100100100 6433 VecMDot 2064 1.0 7.0200e+00 1.6 4.81e+08 1.0 0.0e+00 0.0e+00 2.1e+03 4 6 0 0 9 17 11 0 0 26 3245 VecNorm 4148 1.0 1.0233e+01 1.8 1.95e+08 1.0 0.0e+00 0.0e+00 4.1e+03 6 2 0 0 18 24 5 0 0 52 901 VecScale 3880 1.0 2.1841e-01 2.6 8.19e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 17724 VecCopy 2074 1.0 2.7166e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 8519 1.0 4.5739e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 2843 1.0 2.7114e-01 1.4 1.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 3 0 0 0 22680 VecAYPX 1037 1.0 1.6162e-01 1.5 3.19e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 9340 VecWAXPY 5 1.0 2.2409e-03 1.7 2.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6169 VecMAXPY 4133 1.0 2.3573e+00 1.6 9.75e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 11 0 0 0 6 23 0 0 0 19578 VecAssemblyBegin 5 1.0 3.8697e-02 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 1.5497e-05 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 5428 1.0 6.7515e-01 2.6 0.00e+00 0.0 1.6e+06 2.9e+03 0.0e+00 0 0 43 30 0 1 0100100 0 0 VecScatterEnd 5428 1.0 4.9356e+00 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 10 0 0 0 0 0 VecNormalize 3612 1.0 7.7312e+00 1.7 1.99e+08 1.0 0.0e+00 0.0e+00 3.6e+03 5 2 0 0 16 18 5 0 0 46 1215 MatMult 3880 1.0 9.8883e+00 1.5 1.33e+09 1.0 1.3e+06 3.5e+03 0.0e+00 7 15 34 29 0 27 32 79 96 0 6334 MatMultAdd 774 1.0 1.3812e+00 3.1 3.30e+07 1.0 1.7e+05 5.0e+02 0.0e+00 1 0 4 1 0 3 1 10 2 0 1133 MatMultTranspose 774 1.0 1.0163e+00 1.9 3.30e+07 1.0 1.7e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1540 MatSolve 3612 1.2 4.1856e+00 1.6 9.42e+08 1.1 0.0e+00 0.0e+00 0.0e+00 3 10 0 0 0 11 22 0 0 0 10208 MatLUFactorSym 5 1.0 5.9757e-0368.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 20 1.0 9.2276e-02 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4412 MatILUFactorSym 15 1.0 4.9916e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 20 1.3 4.3893e-0476.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 20 1.3 2.0333e-02 6.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 20 1.0 1.3867e-01 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2936 PCSetUpOnBlocks 1806 1.0 1.4069e-01 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2894 PCApply 258 1.0 2.1168e+01 1.0 2.46e+09 1.1 1.5e+06 2.3e+03 5.5e+03 17 28 40 22 24 68 58 94 73 69 5404 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Container 0 10 5480 0 Krylov Solver 0 91 970112 0 Vector 11 1172 387333592 0 Vector Scatter 0 114 118104 0 Matrix 0 310 190831360 0 Distributed Mesh 0 22 16761112 0 Bipartite Graph 0 44 30096 0 Index Set 0 140 2717000 0 IS L to G Mapping 0 33 11117700 0 Preconditioner 0 90 89240 0 --- Event Stage 1: DMMG Setup Krylov Solver 1 0 0 0 Vector 5 4 5856 0 Vector Scatter 4 0 0 0 Distributed Mesh 2 0 0 0 Bipartite Graph 4 0 0 0 Index Set 10 10 511452 0 IS L to G Mapping 3 0 0 0 --- Event Stage 2: Pressure RHS Setup Container 5 0 0 0 Krylov Solver 60 15 451800 0 Vector 900 610 65595960 0 Vector Scatter 115 60 62160 0 Matrix 355 220 226589740 0 Matrix Coarsen 15 15 9180 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 280 270 3224240 0 IS L to G Mapping 15 0 0 0 Preconditioner 60 15 12360 0 Viewer 1 0 0 0 PetscRandom 15 15 9120 0 --- Event Stage 3: Pressure Solve Vector 782 492 230472480 0 Matrix 20 0 0 0 Index Set 60 0 0 0 --- Event Stage 4: Corrector RHS Setup Container 5 0 0 0 Krylov Solver 60 15 451800 0 Vector 900 610 65595960 0 Vector Scatter 115 60 62160 0 Matrix 355 220 226589740 0 Matrix Coarsen 15 15 9180 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 280 270 3224240 0 IS L to G Mapping 15 0 0 0 Preconditioner 60 15 12360 0 PetscRandom 15 15 9120 0 --- Event Stage 5: Corrector Solve Vector 816 526 246399440 0 Matrix 20 0 0 0 Index Set 60 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000296211 Average time for zero size MPI_Send(): 0.000285755 #PETSc Option Table entries: -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1.0e-7 -ksp_type fgmres -log_summary -mg_levels_ksp_max_it 1 -mg_levels_ksp_type gmres -mg_levels_pc_type bjacobi -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Tue Sep 18 16:22:39 2012 Configure options: --with-debugging=0 --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich ----------------------------------------- Libraries compiled on Tue Sep 18 16:22:39 2012 on firefox.bioinfo.ittc.ku.edu Machine characteristics: Linux-2.6.18-92.1.13.el5-x86_64-with-redhat-5.2-Final Using PETSc directory: /bio/work1/zlwei/PETSc/petsc-dev Using PETSc arch: arch-linux2-c-debug ----------------------------------------- Using C compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include ----------------------------------------- Using C linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc Using Fortran linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lgcc_s -ldl ----------------------------------------- From jedbrown at mcs.anl.gov Fri Oct 5 14:42:56 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 5 Oct 2012 14:42:56 -0500 Subject: [petsc-users] A similar question of "Enquiry regarding log summary results" In-Reply-To: <506F35FE.9070709@gmail.com> References: <506F1003.80105@gmail.com> <506F35FE.9070709@gmail.com> Message-ID: On Fri, Oct 5, 2012 at 2:33 PM, Zhenglun (Alan) Wei wrote: > Dear Dr. Brown, > Thanks for your reply. I checked my grid partition by output the > corner points of the local domain. There is no problem according to that. > I ran the same case with smaller grid size with 8 cores and find that > the code occupy a lot of memory and slow if I use the original script file > which is > *mpiexec -np 8 ./ex45 -pc_type exotic -ksp_type fgmres > -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 1 -mg_levels_pc_type > bjacobi -ksp_rtol 1.0e-7 -ksp_monitor_true_residual -ksp_converged_reason > -log_summary > out* > > If I simply use > *mpiexec -np 8 ./ex45 -ksp_monitor_true_residual -ksp_converged_reason > -log_summary > out* > > Everything works fine. Therefore, I did more tests and found out that > the "-pc_type exotic" has some problem. > PCEXOTIC is a special purpose, and mostly experimental, method. > Now I changed it to "-pc_type gamg"; like, > *mpiexec -np 8 ./ex45 -pc_type gamg -ksp_type fgmres -mg_levels_ksp_type > gmres -mg_levels_ksp_max_it 1 -mg_levels_pc_type bjacobi -ksp_rtol 1.0e-7 > -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out* > Add -pc_gamg_agg_nsmooths 1 to the options above and it will likely converge much faster (in exchange for somewhat higher setup, but you are spending almost no time in setup). Why are you using GMRES on levels? It will probably go faster if you drop all the -mg_levels_* options above (which will use Chebyshev and SOR, should converge fast, but has higher setup), perhaps going with -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson which will probably also works fine without the setup cost. > * * > it is super fast now, the log file is attached. However, I have > another question that it always show that "unpreconditioned " although I > already put pc_type over there. > " 0 KSP unpreconditioned resid norm 1.800295456670e+01 true resid > norm 1.800295456670e+01 ||r(i)||/||b|| 1.000000000000e+00" > > However, if I changed the 'ksp_type' from 'fgmres' to 'cg', it would > give me: > "0 KSP preconditioned resid norm 1.117293929796e+02 true resid norm > 1.800293604638e+01 ||r(i)||/||b|| 1.000000000000e+00" > This "preconditioned" versus "unpreconditioned" is reporting the norm used by the method, not whether preconditioning is being used. Note that you can only use CG (or normal GMRES) if the preconditioner is linear. Changing/removing the option -mg_levels_ksp_type gmres will make your method linear. Note that you can use normal GMRES in unpreconditioned norm (-ksp_norm_type unpreconditioned) so you don't have to pay for the extra memory and work done by FGMRES. > > The finial question: what are the choice for the *'*-mg_levels_ksp_type'. > Are they as the same as 'ksp_type'? > Yes, you can also tell this via -help. Usually you want to use a stationary method in the smoother, typically chebyshev or richardson. A "Krylov" method like GMRES or CG will make the multigrid cycle nonlinear. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Fri Oct 5 14:51:55 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Fri, 05 Oct 2012 14:51:55 -0500 Subject: [petsc-users] A similar question of "Enquiry regarding log summary results" In-Reply-To: References: <506F1003.80105@gmail.com> <506F35FE.9070709@gmail.com> Message-ID: <506F3A5B.7020508@gmail.com> Dear Dr. Brown, Thank you so much for your help. Your reply is really helpful and knowledgeable. sincerely, Alan On 10/5/2012 2:42 PM, Jed Brown wrote: > On Fri, Oct 5, 2012 at 2:33 PM, Zhenglun (Alan) Wei > > wrote: > > Dear Dr. Brown, > Thanks for your reply. I checked my grid partition by output > the corner points of the local domain. There is no problem > according to that. > I ran the same case with smaller grid size with 8 cores and > find that the code occupy a lot of memory and slow if I use the > original script file which is > /mpiexec -np 8 ./ex45 -pc_type exotic -ksp_type fgmres > -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 1 > -mg_levels_pc_type bjacobi -ksp_rtol 1.0e-7 > -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out/ > > If I simply use > /mpiexec -np 8 ./ex45 -ksp_monitor_true_residual > -ksp_converged_reason -log_summary > out/ > > Everything works fine. Therefore, I did more tests and found > out that the "-pc_type exotic" has some problem. > > > PCEXOTIC is a special purpose, and mostly experimental, method. > > Now I changed it to "-pc_type gamg"; like, > /mpiexec -np 8 ./ex45 -pc_type gamg -ksp_type fgmres > -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 1 > -mg_levels_pc_type bjacobi -ksp_rtol 1.0e-7 > -ksp_monitor_true_residual -ksp_converged_reason -log_summary > out/ > > > Add -pc_gamg_agg_nsmooths 1 to the options above and it will likely > converge much faster (in exchange for somewhat higher setup, but you > are spending almost no time in setup). > > Why are you using GMRES on levels? It will probably go faster if you > drop all the -mg_levels_* options above (which will use Chebyshev and > SOR, should converge fast, but has higher setup), perhaps going with > -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson which will > probably also works fine without the setup cost. > > // > it is super fast now, the log file is attached. However, I > have another question that it always show that "unpreconditioned " > although I already put pc_type over there. > " 0 KSP unpreconditioned resid norm 1.800295456670e+01 true > resid norm 1.800295456670e+01 ||r(i)||/||b|| 1.000000000000e+00" > > However, if I changed the 'ksp_type' from 'fgmres' to 'cg', > it would give me: > "0 KSP preconditioned resid norm 1.117293929796e+02 true > resid norm 1.800293604638e+01 ||r(i)||/||b|| 1.000000000000e+00" > > > This "preconditioned" versus "unpreconditioned" is reporting the norm > used by the method, not whether preconditioning is being used. Note > that you can only use CG (or normal GMRES) if the preconditioner is > linear. Changing/removing the option -mg_levels_ksp_type gmres will > make your method linear. Note that you can use normal GMRES in > unpreconditioned norm (-ksp_norm_type unpreconditioned) so you don't > have to pay for the extra memory and work done by FGMRES. > > > The finial question: what are the choice for the > /'/-mg_levels_ksp_type'. Are they as the same as 'ksp_type'? > > > Yes, you can also tell this via -help. Usually you want to use a > stationary method in the smoother, typically chebyshev or richardson. > A "Krylov" method like GMRES or CG will make the multigrid cycle > nonlinear. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Fri Oct 5 19:38:43 2012 From: john.fettig at gmail.com (John Fettig) Date: Fri, 5 Oct 2012 20:38:43 -0400 Subject: [petsc-users] Question about MatMultDiagonalBlock Message-ID: In mpiaij.c, there is this if statement: if (a->inode.size) mat->ops->multdiagonalblock = MatMultDiagonalBlock_MPIAIJ; which prevents the operation MatMultDiagonalBlock from being set if a->inode.size is zero. Why is this? What is wrong with the diagonal block when inode.size==0? Thanks, John From bsmith at mcs.anl.gov Sat Oct 6 15:44:46 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 6 Oct 2012 15:44:46 -0500 Subject: [petsc-users] Question about MatMultDiagonalBlock In-Reply-To: References: Message-ID: <48A7BD46-4493-459D-B6BD-24E9B74E9C24@mcs.anl.gov> On Oct 5, 2012, at 7:38 PM, John Fettig wrote: > In mpiaij.c, there is this if statement: > > if (a->inode.size) mat->ops->multdiagonalblock = MatMultDiagonalBlock_MPIAIJ; > > which prevents the operation MatMultDiagonalBlock from being set if > a->inode.size is zero. Why is this? What is wrong with the diagonal > block when inode.size==0? > > Thanks, > John John, When innode.size is zero the "diagonal blocks" are all 1 by 1. So, yes, there is really no theoretical reason we couldn't provide a function for that case. The current MatMultDiagonalBlock_SeqAIJ_Inode() uses the inode data structure to do the computation so could not be used directly in that case. MatMultDiagonalBlock is only used for Eisenstat's trick for AIJ matrices with inodes and when it has no inodes it used VecPointWiseMult() with the diagonal of the matrix. #undef __FUNCT__ #define __FUNCT__ "PCApply_Eisenstat" static PetscErrorCode PCApply_Eisenstat(PC pc,Vec x,Vec y) { PC_Eisenstat *eis = (PC_Eisenstat*)pc->data; PetscErrorCode ierr; PetscBool hasop; PetscFunctionBegin; if (eis->usediag) { ierr = MatHasOperation(pc->pmat,MATOP_MULT_DIAGONAL_BLOCK,&hasop);CHKERRQ(ierr); if (hasop) { ierr = MatMultDiagonalBlock(pc->pmat,x,y);CHKERRQ(ierr); } else { ierr = VecPointwiseMult(y,x,eis->diag);CHKERRQ(ierr); } } else {ierr = VecCopy(x,y);CHKERRQ(ierr);} PetscFunctionReturn(0); } Are you using MatMultDiagonalBlock() for some other case? Barry From zonexo at gmail.com Sat Oct 6 16:19:55 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 06 Oct 2012 23:19:55 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> Message-ID: <5070A07B.3080900@gmail.com> On 5/10/2012 5:17 PM, Jed Brown wrote: > Now most of the time is spent in PCApply, though an inordinate amount > is still in VecScatterEnd, indicating that either your partition is > very bad or the machine has contention (either with other jobs, zombie > processes, incorrect affinity, over-subscribed). > > In any case, you can try changing the smoother. For example, try > skipping the Chebyshev iterations and just use one iteration of SOR... May I know where can I find all the options for GAMG? I can't find it in the manual. I also used -info for the GAMG and what does these mean: 1. [9] MatAssemblyBegin_MPIAIJ(): Stash has 14396 entries, uses 1 mallocs How serious is this? Does it mean there's a lot of mallocs going on? Do I correct this by using higher d_nz and o_nz? 2. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 112785 X 112785; storage space: 0 unneeded,3628127 used [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 When I use MatCreateAIJ, my d_nz and o_nz are both 7. If the row is 59, does it mean that my d_nz and o_nz are too small. Must I set it to the maximum nonzeros given by -info? 3. [0] MatSetOption_SeqAIJ(): Option STRUCTURALLY_SYMMETRIC ignored My Poisson matrix is symmetrical, so why is it ignored? > > On Fri, Oct 5, 2012 at 10:08 AM, TAY wee-beng > wrote: > > On 4/10/2012 11:44 PM, Satish Balay wrote: > > On Thu, 4 Oct 2012, Matthew Knepley wrote: > > the Mom-Z solve is bad because the assembly of the operator is > screwed up. You are communicating too many values. So, > jsut go into > your code and count how many off process entries you set. > > running with '-info' should give debug info regarding this > communication [stack size used for messages, and the number of > mallocs > needed etc..] > > Satish > > > Ok, I found my problem. I was using: > > PetscLogStage stage > > and the time for the poisson eqn got combined into that of the > momentum_z solve. Hence the confusion in results. > > I have attached the new results. The time is taken up mainly by > GAMG. I am using: > > -poisson_pc_gamg_agg_nsmooths 1 > -poisson_pc_type gamg > > What are the possible ways to improve the efficiency or reduce the > time? > > If I use HYPRE's geometric multigrid, performance is somewhat > better, such that momentum_x/y/z accounts for a total of 15% > computing time. > > From the log_summary, for the momentum eqns, it seems that > MatAssembly is the main problem, is that so? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.fettig at gmail.com Sat Oct 6 16:30:29 2012 From: john.fettig at gmail.com (John Fettig) Date: Sat, 6 Oct 2012 17:30:29 -0400 Subject: [petsc-users] Question about MatMultDiagonalBlock In-Reply-To: <48A7BD46-4493-459D-B6BD-24E9B74E9C24@mcs.anl.gov> References: <48A7BD46-4493-459D-B6BD-24E9B74E9C24@mcs.anl.gov> Message-ID: On Sat, Oct 6, 2012 at 4:44 PM, Barry Smith wrote: > On Oct 5, 2012, at 7:38 PM, John Fettig wrote: >>What is wrong with the diagonal >> block when inode.size==0? >> >> Thanks, >> John > > John, > > When innode.size is zero the "diagonal blocks" are all 1 by 1. ... > Are you using MatMultDiagonalBlock() for some other case? Barry, Thanks for the explanation, it makes sense now. I was trying to use it in the context of a multigrid, and didn't think about the coarse level. It must have been failing when the diagonal block was 1. In the end, though, I ended up not using this function. I was just curious why there should be any restriction. John From jedbrown at mcs.anl.gov Sat Oct 6 16:42:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 6 Oct 2012 16:42:09 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <5070A07B.3080900@gmail.com> References: <5064051D.1060903@gmail.com> <5068B903.8060400@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> <5070A07B.3080900@gmail.com> Message-ID: On Sat, Oct 6, 2012 at 4:19 PM, TAY wee-beng wrote: > May I know where can I find all the options for GAMG? I can't find it in > the manual. > As always, run with -help. The options for GAMG specifically are controlled under the prefix -pc_gamg_*. The smoothers and coarse grid solvers are configured under -mg_levels_*. > > I also used -info for the GAMG and what does these mean: > > 1. [9] MatAssemblyBegin_MPIAIJ(): Stash has 14396 entries, uses 1 mallocs > > How serious is this? Does it mean there's a lot of mallocs going on? Do I > correct this by using higher d_nz and o_nz? > No, the stash holds entries that are stored on a different process from where you assemble. Assembling some values off-process is common for finite element discretizations, for example. Less than one percent of the values are assembled off-process, which is fine. > > 2. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 112785 X 112785; storage > space: 0 unneeded,3628127 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 > > When I use MatCreateAIJ, my d_nz and o_nz are both 7. If the row is 59, > does it mean that my d_nz and o_nz are too small. Must I set it to the > maximum nonzeros given by -info? > Either this line is not the first assembly or you didn't pass 7. If you don't want to use extra memory, you should preallocate exactly, otherwise it is better to over-estimate than to under-estimate. > > 3. [0] MatSetOption_SeqAIJ(): Option STRUCTURALLY_SYMMETRIC ignored > > My Poisson matrix is symmetrical, so why is it ignored? > This is a bad info message. That option is handled at a higher level. I'll fix it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sat Oct 6 16:51:42 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sat, 06 Oct 2012 23:51:42 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> <5070A07B.3080900@gmail.com> Message-ID: <5070A7EE.4060408@gmail.com> On 6/10/2012 11:42 PM, Jed Brown wrote: > On Sat, Oct 6, 2012 at 4:19 PM, TAY wee-beng > wrote: > > May I know where can I find all the options for GAMG? I can't find > it in the manual. > > > As always, run with -help. The options for GAMG specifically are > controlled under the prefix -pc_gamg_*. The smoothers and coarse grid > solvers are configured under -mg_levels_*. > > > I also used -info for the GAMG and what does these mean: > > 1. [9] MatAssemblyBegin_MPIAIJ(): Stash has 14396 entries, uses 1 > mallocs > > > How serious is this? Does it mean there's a lot of mallocs going > on? Do I correct this by using higher d_nz and o_nz? > > > No, the stash holds entries that are stored on a different process > from where you assemble. Assembling some values off-process is common > for finite element discretizations, for example. Less than one percent > of the values are assembled off-process, which is fine. > > > 2. [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 112785 X 112785; > storage space: 0 unneeded,3628127 used > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during > MatSetValues() is 0 > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 59 > > When I use MatCreateAIJ, my d_nz and o_nz are both 7. If the row > is 59, does it mean that my d_nz and o_nz are too small. Must I > set it to the maximum nonzeros given by -info? > > > Either this line is not the first assembly or you didn't pass 7. If > you don't want to use extra memory, you should preallocate exactly, > otherwise it is better to over-estimate than to under-estimate. This line happens rather late in the assembly, so it is not the main matrix which I've created, right? And hence this doesn't matter? > > > 3. [0] MatSetOption_SeqAIJ(): Option STRUCTURALLY_SYMMETRIC ignored > > My Poisson matrix is symmetrical, so why is it ignored? > > > This is a bad info message. That option is handled at a higher level. > I'll fix it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 6 16:54:24 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 6 Oct 2012 16:54:24 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <5070A7EE.4060408@gmail.com> References: <5064051D.1060903@gmail.com> <506ADFAB.8080006@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> <5070A07B.3080900@gmail.com> <5070A7EE.4060408@gmail.com> Message-ID: On Sat, Oct 6, 2012 at 4:51 PM, TAY wee-beng wrote: > This line happens rather late in the assembly, so it is not the main > matrix which I've created, right? And hence this doesn't matter? Look at all the lines talking about mallocs. If the first assembly uses lots of mallocs, the first assembly will be slow. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Sat Oct 6 20:04:57 2012 From: zhenglun.wei at gmail.com (Alan) Date: Sat, 06 Oct 2012 20:04:57 -0500 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement Message-ID: <5070D539.905@gmail.com> Dear Folks, I hope you're having a nice day. I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to obtain solutions (Poisson Equation) with local refined grid from HYPRE. Is there any examples in PETSc that I can refer? or is there any examples in PETSc which introduce such a way that the Matrix is imported from external software rather than generated by PETSc. thanks, Alan From jedbrown at mcs.anl.gov Sat Oct 6 20:34:13 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 6 Oct 2012 20:34:13 -0500 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <5070D539.905@gmail.com> References: <5070D539.905@gmail.com> Message-ID: Run any example with -pc_type hypre. I'm not sure what you're asking for. On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: > Dear Folks, > I hope you're having a nice day. > I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to > obtain solutions (Poisson Equation) with local refined grid from HYPRE. > Is there any examples in PETSc that I can refer? or is there any > examples in PETSc which introduce such a way that the Matrix is imported > from external software rather than generated by PETSc. > > thanks, > Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Sat Oct 6 21:47:49 2012 From: zhenglun.wei at gmail.com (Alan) Date: Sat, 06 Oct 2012 21:47:49 -0500 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: References: <5070D539.905@gmail.com> Message-ID: <5070ED55.8060709@gmail.com> Dear Dr. Brown, Thanks for your reply and sorry for confusing you. I was trying to ask if there is an alternative way that I can replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with something from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a 3D Cartesian grid with uniform grid size. However, I'm trying to import a local refined grid from outside of the PETSc (i.e. HYPRE) to replace this 3D Cartesian grid with unifrom grid size. Is there any examples for me to refer? thanks, Alan > Run any example with -pc_type hypre. I'm not sure what you're asking for. > > On Sat, Oct 6, 2012 at 8:04 PM, Alan > wrote: > > Dear Folks, > I hope you're having a nice day. > I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to > obtain solutions (Poisson Equation) with local refined grid from > HYPRE. > Is there any examples in PETSc that I can refer? or is there any > examples in PETSc which introduce such a way that the Matrix is > imported > from external software rather than generated by PETSc. > > thanks, > Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Oct 6 21:50:40 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 6 Oct 2012 22:50:40 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <5070ED55.8060709@gmail.com> References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> Message-ID: On Sat, Oct 6, 2012 at 10:47 PM, Alan wrote: > Dear Dr. Brown, > Thanks for your reply and sorry for confusing you. > I was trying to ask if there is an alternative way that I can replace > the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with something > from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a 3D > Cartesian grid with uniform grid size. However, I'm trying to import a > local refined grid from outside of the PETSc (i.e. HYPRE) to replace this > 3D Cartesian grid with unifrom grid size. Is there any examples for me to > refer? > Do we have automatic support for this? No. You could of course just use the Vec/Mat interface, but it sounds like a lot of coding on your part. I plan to do this eventually by specifying the data layout over the grid using PetscSection. However, the really important question here is, what do you need from the grid? If you don't need topological queries, just make the Vec/Mat and be done with it. If you do (which I imagine), then you need to be specific about what you need. Matt > thanks, > Alan > > Run any example with -pc_type hypre. I'm not sure what you're asking for. > > On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: > >> Dear Folks, >> I hope you're having a nice day. >> I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to >> obtain solutions (Poisson Equation) with local refined grid from HYPRE. >> Is there any examples in PETSc that I can refer? or is there any >> examples in PETSc which introduce such a way that the Matrix is imported >> from external software rather than generated by PETSc. >> >> thanks, >> Alan >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhenglun.wei at gmail.com Sun Oct 7 11:59:20 2012 From: zhenglun.wei at gmail.com (Alan) Date: Sun, 07 Oct 2012 11:59:20 -0500 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> Message-ID: <5071B4E8.6050003@gmail.com> Dear Dr. Brown, I was trying to use PETSc to mimic something like Chombo. The figure in this website represents my thought very well. (https://commons.lbl.gov/display/chombo/Chombo+-+Software+for+Adaptive+Solutions+of+Partial+Differential+Equations). My code simulates a flow over a objective, i.e. 2D cylinder. Finer resolution is needed near the objective. I can manually define where the finer resolution is needed. BTW, I wonder if there is any examples for the Vec/Mat interface. Also, is there any examples for PETSc for GPU. thanks, Alan > On Sat, Oct 6, 2012 at 10:47 PM, Alan > wrote: > > Dear Dr. Brown, > Thanks for your reply and sorry for confusing you. > I was trying to ask if there is an alternative way that I can > replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c > with something from outside of the PETSc (i.e. HYPRE). > DMDACreate3d generates a 3D Cartesian grid with uniform grid size. > However, I'm trying to import a local refined grid from outside of > the PETSc (i.e. HYPRE) to replace this 3D Cartesian grid with > unifrom grid size. Is there any examples for me to refer? > > > Do we have automatic support for this? No. You could of course just > use the Vec/Mat interface, but > it sounds like a lot of coding on your part. I plan to do this > eventually by specifying the data layout over > the grid using PetscSection. However, the really important question > here is, what do you need from the > grid? If you don't need topological queries, just make the Vec/Mat and > be done with it. If you do (which > I imagine), then you need to be specific about what you need. > > Matt > > thanks, > Alan > >> Run any example with -pc_type hypre. I'm not sure what you're >> asking for. >> >> On Sat, Oct 6, 2012 at 8:04 PM, Alan > > wrote: >> >> Dear Folks, >> I hope you're having a nice day. >> I'm trying to couple HYPRE with PETSc in order to use PETSc >> solvers to >> obtain solutions (Poisson Equation) with local refined grid >> from HYPRE. >> Is there any examples in PETSc that I can refer? or is there any >> examples in PETSc which introduce such a way that the Matrix >> is imported >> from external software rather than generated by PETSc. >> >> thanks, >> Alan >> >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Sun Oct 7 16:45:04 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Sun, 07 Oct 2012 23:45:04 +0200 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: References: <5064051D.1060903@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> <5070A07B.3080900@gmail.com> <5070A7EE.4060408@gmail.com> Message-ID: <5071F7E0.5090103@gmail.com> Hi, I have attached 3 results using 12,24 and 32 processors. I am using a completely different clusters and I'm testing if it's the cluster configuration problems. It's seems that VecScatterEnd does not scale well from 12 to 24 to 32. Does these results show that there's problems with the partition still? My partition is clustered closely at the center, I'm wondering if this has a great effect on scaling... Tks! Yours sincerely, TAY wee-beng On 6/10/2012 11:54 PM, Jed Brown wrote: > On Sat, Oct 6, 2012 at 4:51 PM, TAY wee-beng > wrote: > > This line happens rather late in the assembly, so it is not the > main matrix which I've created, right? And hence this doesn't matter? > > > Look at all the lines talking about mallocs. If the first assembly > uses lots of mallocs, the first assembly will be slow. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_atlas6_rel named atlas6-c39 with 12 processors, by g0306332 Mon Oct 8 03:58:56 2012 Using Petsc Development HG revision: 0b92fc173218fc24fd69e1ee041f811d50d4766c HG Date: Fri Oct 05 15:20:44 2012 -0500 Max Max/Min Avg Total mv: cannot move `/var/tmp/uvwp12.plt' to `/home/wtay/Results/8/': No such file or directory Time (sec): 1.201e+03 1.00258 1.199e+03 Objects: 5.160e+02 1.00000 5.160e+02 Flops: 2.977e+11 1.11376 2.739e+11 3.287e+12 Flops/sec: 2.478e+08 1.11115 2.285e+08 2.742e+09 MPI Messages: 5.879e+04 2.96007 4.346e+04 5.215e+05 MPI Message Lengths: 4.194e+09 2.02856 8.822e+04 4.601e+10 MPI Reductions: 6.296e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.3933e+02 11.6% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+01 0.3% 1: poisson_eqn: 9.8694e+02 82.3% 3.1635e+12 96.3% 5.206e+05 99.8% 8.705e+04 98.7% 5.977e+03 94.9% 2: momentum_eqn: 7.2420e+01 6.0% 1.2324e+11 3.7% 8.800e+02 0.2% 1.170e+03 1.3% 2.980e+02 4.7% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson_eqn MatMult 21340 1.0 5.8555e+02 1.1 1.91e+11 1.2 4.0e+05 1.1e+05 0.0e+00 47 62 77 93 0 58 65 77 94 0 3503 MatMultAdd 3280 1.0 4.0142e+01 1.2 1.26e+10 1.2 5.7e+04 1.9e+04 0.0e+00 3 4 11 2 0 4 4 11 2 0 3205 MatMultTranspose 3280 1.0 4.4090e+01 1.3 1.26e+10 1.2 5.7e+04 1.9e+04 0.0e+00 3 4 11 2 0 4 4 11 2 0 2918 MatSolve 1640 0.0 5.8434e-03 0.0 1.42e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 243 MatLUFactorSym 1 1.0 3.1948e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 1.7881e-05 6.2 5.53e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 309 MatConvert 4 1.0 2.3882e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 2.8265e-01 1.1 5.26e+07 1.2 7.8e+01 1.0e+05 0.0e+00 0 0 0 0 0 0 0 0 0 0 1960 MatAssemblyBegin 69 1.0 1.6842e+00 2.8 0.00e+00 0.0 2.3e+02 1.8e+04 7.4e+01 0 0 0 0 1 0 0 0 0 1 0 MatAssemblyEnd 69 1.0 1.8641e+00 1.0 0.00e+00 0.0 1.0e+03 1.6e+04 2.0e+02 0 0 0 0 3 0 0 0 0 3 0 MatGetRow 3544348 1.0 4.4062e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 0.0 6.9141e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 4.5061e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.7e-01 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 6.5285e-01 1.2 0.00e+00 0.0 4.3e+02 1.4e+05 9.2e+01 0 0 0 0 1 0 0 0 0 2 0 MatAXPY 4 1.0 6.9771e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 1.3333e+00 1.0 3.72e+07 1.2 4.9e+02 4.9e+04 9.6e+01 0 0 0 0 2 0 0 0 0 2 298 MatMatMultSym 4 1.0 1.0190e+00 1.0 0.00e+00 0.0 4.1e+02 3.9e+04 8.8e+01 0 0 0 0 1 0 0 0 0 1 0 MatMatMultNum 4 1.0 3.1801e-01 1.0 3.72e+07 1.2 7.8e+01 1.0e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 1249 MatPtAP 4 1.0 4.5583e+00 1.0 1.16e+09 1.8 8.4e+02 1.1e+05 1.1e+02 0 0 0 0 2 0 0 0 0 2 1881 MatPtAPSymbolic 4 1.0 2.8222e+00 1.0 0.00e+00 0.0 7.5e+02 9.3e+04 1.0e+02 0 0 0 0 2 0 0 0 0 2 0 MatPtAPNumeric 4 1.0 1.7362e+00 1.0 1.16e+09 1.8 8.9e+01 2.7e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 4939 MatTrnMatMult 4 1.0 1.2843e+01 1.0 2.04e+09 1.9 4.7e+02 6.6e+05 1.2e+02 1 1 0 1 2 1 1 0 1 2 1775 MatGetLocalMat 20 1.0 7.1283e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 12 1.0 8.6687e-02 1.6 0.00e+00 0.0 5.5e+02 1.4e+05 1.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 9.0438e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 860 1.0 8.2049e-01 1.1 3.90e+08 1.0 0.0e+00 0.0e+00 8.6e+02 0 0 0 0 14 0 0 0 0 14 5608 KSPSetUp 11 1.0 1.3758e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 20 1.0 9.8386e+02 1.0 2.88e+11 1.1 5.2e+05 8.7e+04 5.8e+03 82 96100 99 93 100100100100 98 3215 VecDot 800 1.0 1.2949e+01 2.7 2.57e+09 1.0 0.0e+00 0.0e+00 8.0e+02 1 1 0 0 13 1 1 0 0 13 2355 VecDotNorm2 400 1.0 1.3116e+01 2.2 5.14e+09 1.0 0.0e+00 0.0e+00 1.2e+03 1 2 0 0 19 1 2 0 0 20 4650 VecMDot 860 1.0 4.3056e-01 1.2 1.95e+08 1.0 0.0e+00 0.0e+00 8.6e+02 0 0 0 0 14 0 0 0 0 14 5343 VecNorm 2104 1.0 1.0177e+01 5.3 1.39e+09 1.0 0.0e+00 0.0e+00 2.1e+03 0 1 0 0 33 0 1 0 0 35 1618 VecScale 14804 1.0 2.2892e+01 1.1 5.83e+09 1.0 0.0e+00 0.0e+00 0.0e+00 2 2 0 0 0 2 2 0 0 0 3007 VecCopy 4144 1.0 7.4420e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecSet 14075 1.0 6.1308e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 27064 1.0 7.9051e+01 1.2 2.33e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 8 0 0 0 8 9 0 0 0 3472 VecAYPX 26240 1.0 7.8797e+01 1.1 1.45e+10 1.0 0.0e+00 0.0e+00 0.0e+00 6 5 0 0 0 8 5 0 0 0 2176 VecAXPBYCZ 800 1.0 1.2601e+01 1.0 5.14e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 4840 VecWAXPY 800 1.0 1.2598e+01 1.0 2.57e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 2420 VecMAXPY 1684 1.0 4.7353e-01 1.0 2.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5742 VecAssemblyBegin 87 1.0 4.7785e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.6e+02 0 0 0 0 4 0 0 0 0 4 0 VecAssemblyEnd 87 1.0 9.9659e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 19724 1.0 8.5318e+01 1.1 8.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00 7 3 0 0 0 8 3 0 0 0 1209 VecScatterBegin 27955 1.0 3.7788e+00 2.3 0.00e+00 0.0 5.2e+05 8.7e+04 0.0e+00 0 0 99 98 0 0 0100 99 0 0 VecScatterEnd 27955 1.0 6.0687e+01 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecSetRandom 4 1.0 2.6621e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 1684 1.0 8.1183e+0052.0 5.86e+07 1.0 0.0e+00 0.0e+00 1.7e+03 0 0 0 0 27 0 0 0 0 28 85 PCSetUp 2 1.0 2.7046e+01 1.0 3.60e+09 1.1 4.4e+03 1.6e+05 9.6e+02 2 1 1 1 15 3 1 1 2 16 1555 PCSetUpOnBlocks 820 1.0 6.9976e-04 1.3 5.53e+03 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 8 PCApply 820 1.0 8.5955e+02 1.0 2.52e+11 1.1 5.0e+05 8.1e+04 2.5e+03 71 83 96 88 39 86 86 96 89 41 3171 PCGAMGgraph_AGG 4 1.0 2.9757e+00 1.0 3.72e+07 1.2 2.3e+02 5.1e+04 7.6e+01 0 0 0 0 1 0 0 0 0 1 133 PCGAMGcoarse_AGG 4 1.0 1.4145e+01 1.0 2.04e+09 1.9 1.3e+03 3.4e+05 2.9e+02 1 1 0 1 5 1 1 0 1 5 1612 PCGAMGProl_AGG 4 1.0 1.0395e+00 1.0 0.00e+00 0.0 5.5e+02 6.4e+04 1.1e+02 0 0 0 0 2 0 0 0 0 2 0 PCGAMGPOpt_AGG 4 1.0 4.1134e+00 1.0 9.11e+08 1.1 1.3e+03 8.0e+04 2.1e+02 0 0 0 0 3 0 0 0 0 4 2503 --- Event Stage 2: momentum_eqn MatMult 38 1.0 8.5567e+00 1.0 2.35e+09 1.0 8.4e+02 7.2e+05 0.0e+00 1 1 0 1 0 12 23 95 99 0 3260 MatSolve 57 1.0 1.3324e+01 1.1 3.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 18 34 0 0 0 3123 MatLUFactorNum 19 1.0 1.3298e+01 1.1 1.97e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 18 19 0 0 0 1753 MatILUFactorSym 1 1.0 5.8045e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatAssemblyBegin 19 1.0 3.3103e+0051.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 3 0 0 0 13 0 MatAssemblyEnd 19 1.0 4.1241e+00 1.0 0.00e+00 0.0 4.4e+01 1.8e+05 8.0e+00 0 0 0 0 0 6 0 5 1 3 0 MatGetRowIJ 1 1.0 1.9073e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 5.7246e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 1 0 KSPSetUp 38 1.0 1.5740e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 19 1.0 4.4298e+01 1.0 1.04e+10 1.0 8.4e+02 7.2e+05 1.4e+02 4 4 0 1 2 61100 95 99 46 2782 VecDot 38 1.0 1.3129e+00 1.6 3.66e+08 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 1 4 0 0 13 3310 VecDotNorm2 19 1.0 1.3210e+00 1.6 7.32e+08 1.0 0.0e+00 0.0e+00 5.7e+01 0 0 0 0 1 1 7 0 0 19 6579 VecNorm 38 1.0 1.5438e+00 4.7 3.66e+08 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 1 4 0 0 13 2815 VecCopy 38 1.0 8.6146e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 115 1.0 2.0613e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 0 0 0 0 0 VecAXPBYCZ 38 1.0 1.8039e+00 1.0 7.32e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 4818 VecWAXPY 38 1.0 1.8099e+00 1.0 3.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 4 0 0 0 2401 VecAssemblyBegin 38 1.0 1.0454e-0116.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 2 0 0 0 0 38 0 VecAssemblyEnd 38 1.0 4.7684e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 38 1.0 7.1190e-02 2.4 0.00e+00 0.0 8.4e+02 7.2e+05 0.0e+00 0 0 0 1 0 0 0 95 99 0 0 VecScatterEnd 38 1.0 2.8875e-01 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 38 1.0 1.3929e+01 1.1 1.97e+09 1.0 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 19 19 0 0 2 1674 PCSetUpOnBlocks 19 1.0 1.3929e+01 1.1 1.97e+09 1.0 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 19 19 0 0 1 1674 PCApply 57 1.0 1.4274e+01 1.1 3.51e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 19 34 0 0 0 2915 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 6 48 2093863816 0 Krylov Solver 2 9 27392 0 Vector 4 63 485882816 0 Vector Scatter 0 10 10360 0 Index Set 0 10 19278416 0 Preconditioner 0 9 8868 0 Viewer 1 0 0 0 --- Event Stage 1: poisson_eqn Matrix 117 76 1488820380 0 Matrix Coarsen 4 4 2448 0 Krylov Solver 10 4 120512 0 Vector 225 175 387306216 0 Vector Scatter 31 22 22792 0 Index Set 81 74 807916 0 Preconditioner 11 4 3424 0 PetscRandom 4 4 2432 0 --- Event Stage 2: momentum_eqn Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1496 0 Vector Scatter 1 0 0 0 Index Set 5 2 361680 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 9.20296e-06 Average time for zero size MPI_Send(): 2.16564e-06 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Sun Oct 7 16:51:24 2012 Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --with-mpi-dir=/app1/mvapich2/current/ --with-blas-lapack-dir=/app1/intel/mkl/lib/intel64/ --with-batch=1 --with-debugging=0 --download-hypre=1 --prefix=/home/svu/g0306332/lib/petsc-3.3-dev_atlas6_rel --known-mpi-shared-libraries=0 ----------------------------------------- Libraries compiled on Sun Oct 7 16:51:24 2012 on atlas6-c01 Machine characteristics: Linux-2.6.18-274.7.1.el5-x86_64-with-redhat-5.8-Final Using PETSc directory: /home/svu/g0306332/codes/petsc-dev Using PETSc arch: petsc-3.3-dev_atlas6_rel ----------------------------------------- Using C compiler: /app1/mvapich2/current/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /app1/mvapich2/current/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/app1/mvapich2/current/include -I/app1/mvapich2/1.4/include ----------------------------------------- Using C linker: /app1/mvapich2/current/bin/mpicc Using Fortran linker: /app1/mvapich2/current/bin/mpif90 Using libraries: -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lHYPRE -L/app1/mvapich2/1.4/lib -L/nfs/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/ipp/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichcxx -lstdc++ -lpthread -Wl,-rpath,/app1/intel/mkl/lib/intel64 -L/app1/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpichf90 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl ----------------------------------------- Job /app1/common/lsf7/7.0/linux2.6-glibc2.3-x86_64/bin/mvapich_wrapper ./a.out -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg -log_summary TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME ===== ========== ================ ======================= =================== 00000 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00001 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00002 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00003 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00004 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00005 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00006 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00007 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00008 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00009 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00010 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 00011 atlas6-c39 ./a.out -poisson Done 10/08/2012 03:58:59 -------------- next part -------------- ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_atlas6_rel named atlas6-c14 with 24 processors, by g0306332 Mon Oct 8 04:17:33 2012 Using Petsc Development HG revision: 0b92fc173218fc24fd69e1ee041f811d50d4766c HG Date: Fri Oct 05 15:20:44 2012 -0500 Max Max/Min Avg Total mv: cannot move `/var/tmp/uvwp24.plt' to `/home/wtay/Results/8/': No such file or directory Time (sec): 5.824e+02 1.00339 5.807e+02 Objects: 5.160e+02 1.00000 5.160e+02 Flops: 1.380e+11 1.23808 1.148e+11 2.756e+12 Flops/sec: 2.370e+08 1.23418 1.978e+08 4.746e+09 MPI Messages: 6.066e+04 3.66281 3.892e+04 9.341e+05 MPI Message Lengths: 3.500e+09 2.02046 8.586e+04 8.021e+10 MPI Reductions: 5.548e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.1861e+02 20.4% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+01 0.4% 1: poisson_eqn: 4.2596e+02 73.4% 2.6333e+12 95.5% 9.323e+05 99.8% 8.450e+04 98.4% 5.229e+03 94.3% 2: momentum_eqn: 3.6084e+01 6.2% 1.2276e+11 4.5% 1.840e+03 0.2% 1.366e+03 1.6% 2.980e+02 5.4% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson_eqn MatMult 17700 1.0 2.6405e+02 1.2 9.02e+10 1.3 7.1e+05 1.0e+05 0.0e+00 42 62 76 92 0 58 65 76 94 0 6455 MatMultAdd 2720 1.0 1.9278e+01 1.5 6.25e+09 1.5 1.1e+05 1.7e+04 0.0e+00 3 4 11 2 0 3 4 11 2 0 5553 MatMultTranspose 2720 1.0 2.1702e+01 1.5 6.25e+09 1.5 1.1e+05 1.7e+04 0.0e+00 3 4 11 2 0 4 4 11 2 0 4933 MatSolve 1360 0.0 4.2832e-03 0.0 1.30e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 303 MatLUFactorSym 1 1.0 3.8147e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 2.1935e-05 7.7 6.48e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 295 MatConvert 4 1.0 1.2682e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.4481e-01 1.1 3.05e+07 1.4 1.7e+02 9.8e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 3835 MatAssemblyBegin 69 1.0 2.6819e+00 2.5 0.00e+00 0.0 5.3e+02 1.6e+04 7.4e+01 0 0 0 0 1 0 0 0 0 1 0 MatAssemblyEnd 69 1.0 1.0519e+00 1.0 0.00e+00 0.0 2.6e+03 1.3e+04 2.0e+02 0 0 0 0 4 0 0 0 0 4 0 MatGetRow 1821330 1.1 2.2507e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 0.0 8.1062e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 5.1022e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.3e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 3.9578e-01 1.3 0.00e+00 0.0 3.1e+03 4.7e+04 1.8e+02 0 0 0 0 3 0 0 0 0 4 0 MatAXPY 4 1.0 3.4627e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 7.5009e-01 1.0 2.13e+07 1.3 1.1e+03 4.4e+04 9.6e+01 0 0 0 0 2 0 0 0 0 2 530 MatMatMultSym 4 1.0 5.8047e-01 1.0 0.00e+00 0.0 9.7e+02 3.5e+04 8.8e+01 0 0 0 0 2 0 0 0 0 2 0 MatMatMultNum 4 1.0 1.7238e-01 1.0 2.13e+07 1.3 1.7e+02 9.8e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 2308 MatPtAP 4 1.0 3.0428e+00 1.0 8.45e+08 2.7 2.2e+03 9.0e+04 1.1e+02 1 0 0 0 2 1 0 0 0 2 2823 MatPtAPSymbolic 4 1.0 1.8848e+00 1.0 0.00e+00 0.0 2.0e+03 7.5e+04 1.0e+02 0 0 0 0 2 0 0 0 0 2 0 MatPtAPNumeric 4 1.0 1.1581e+00 1.0 8.45e+08 2.7 2.4e+02 2.1e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 7418 MatTrnMatMult 4 1.0 6.8553e+00 1.0 1.03e+09 3.2 1.1e+03 5.9e+05 1.2e+02 1 1 0 1 2 2 1 0 1 2 3304 MatGetLocalMat 20 1.0 3.6288e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 12 1.0 7.6919e-02 1.6 0.00e+00 0.0 1.2e+03 1.4e+05 1.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 4.5255e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 720 1.0 4.8121e-01 1.2 2.00e+08 1.1 0.0e+00 0.0e+00 7.2e+02 0 0 0 0 13 0 0 0 0 14 9566 KSPSetUp 11 1.0 7.0659e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 20 1.0 4.2393e+02 1.0 1.33e+11 1.3 9.3e+05 8.5e+04 5.1e+03 73 96100 98 92 100100100100 98 6212 VecDot 660 1.0 1.2440e+01 5.9 1.08e+09 1.0 0.0e+00 0.0e+00 6.6e+02 1 1 0 0 12 1 1 0 0 13 2022 VecDotNorm2 330 1.0 1.2494e+01 4.8 2.16e+09 1.0 0.0e+00 0.0e+00 9.9e+02 1 2 0 0 18 2 2 0 0 19 4027 VecMDot 720 1.0 2.8871e-01 1.4 1.00e+08 1.1 0.0e+00 0.0e+00 7.2e+02 0 0 0 0 13 0 0 0 0 14 7972 VecNorm 1754 1.0 1.3658e+0119.0 5.93e+08 1.0 0.0e+00 0.0e+00 1.8e+03 1 1 0 0 32 1 1 0 0 34 1011 VecScale 12284 1.0 8.2996e+00 1.2 2.49e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 2 2 0 0 0 6885 VecCopy 3444 1.0 3.1909e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecSet 11695 1.0 2.7659e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 22444 1.0 3.1521e+01 1.3 9.91e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 8 0 0 0 7 9 0 0 0 7223 VecAYPX 21760 1.0 3.1916e+01 1.2 6.19e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 5 0 0 0 7 5 0 0 0 4458 VecAXPBYCZ 660 1.0 5.2597e+00 1.1 2.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 9566 VecWAXPY 660 1.0 5.2671e+00 1.1 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 4776 VecMAXPY 1404 1.0 2.3818e-01 1.1 1.18e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 11420 VecAssemblyBegin 110 1.0 8.9732e-01 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 3.2e+02 0 0 0 0 6 0 0 0 0 6 0 VecAssemblyEnd 110 1.0 1.0490e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 16364 1.0 3.4136e+01 1.2 3.73e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 3 0 0 0 7 3 0 0 0 2507 VecScatterBegin 23218 1.0 3.1578e+00 2.3 0.00e+00 0.0 9.3e+05 8.4e+04 0.0e+00 0 0 99 97 0 1 0 99 99 0 0 VecScatterEnd 23218 1.0 4.9513e+01 5.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 7 0 0 0 0 0 VecSetRandom 4 1.0 1.3620e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 1404 1.0 1.2207e+01120.4 3.01e+07 1.1 0.0e+00 0.0e+00 1.4e+03 1 0 0 0 25 1 0 0 0 27 57 PCSetUp 2 1.0 1.5744e+01 1.0 1.83e+09 1.2 1.2e+04 1.2e+05 1.1e+03 3 2 1 2 19 4 2 1 2 20 2664 PCSetUpOnBlocks 680 1.0 6.5613e-04 1.6 6.48e+03 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 10 PCApply 680 1.0 3.6728e+02 1.1 1.18e+11 1.3 8.9e+05 7.9e+04 2.1e+03 62 82 95 87 37 84 86 95 89 39 6165 PCGAMGgraph_AGG 4 1.0 1.5585e+00 1.0 2.13e+07 1.3 4.9e+02 4.9e+04 7.6e+01 0 0 0 0 1 0 0 0 0 1 255 PCGAMGcoarse_AGG 4 1.0 7.5525e+00 1.0 1.03e+09 3.2 5.2e+03 1.8e+05 3.8e+02 1 1 1 1 7 2 1 1 1 7 2999 PCGAMGProl_AGG 4 1.0 1.3036e+00 1.0 0.00e+00 0.0 1.3e+03 5.5e+04 1.1e+02 0 0 0 0 2 0 0 0 0 2 0 PCGAMGPOpt_AGG 4 1.0 2.1967e+00 1.0 4.93e+08 1.2 2.8e+03 7.6e+04 2.1e+02 0 0 0 0 4 1 0 0 0 4 4693 --- Event Stage 2: momentum_eqn MatMult 38 1.0 4.4312e+00 1.1 1.20e+09 1.1 1.7e+03 7.2e+05 0.0e+00 1 1 0 2 0 12 23 95 99 0 6295 MatSolve 57 1.0 6.1924e+00 1.1 1.78e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 17 34 0 0 0 6680 MatLUFactorNum 19 1.0 6.1136e+00 1.1 9.93e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 16 19 0 0 0 3776 MatILUFactorSym 1 1.0 2.6631e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatAssemblyBegin 19 1.0 2.0705e+0034.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 4 0 0 0 13 0 MatAssemblyEnd 19 1.0 2.1199e+00 1.0 0.00e+00 0.0 9.2e+01 1.8e+05 8.0e+00 0 0 0 0 0 6 0 5 1 3 0 MatGetRowIJ 1 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.9305e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 1 0 KSPSetUp 38 1.0 8.0219e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 19 1.0 2.1292e+01 1.0 5.28e+09 1.1 1.7e+03 7.2e+05 1.4e+02 4 4 0 2 2 59100 95 99 46 5766 VecDot 38 1.0 7.3666e-01 1.9 1.87e+08 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 1 4 0 0 13 5899 VecDotNorm2 19 1.0 7.0598e-01 1.7 3.73e+08 1.0 0.0e+00 0.0e+00 5.7e+01 0 0 0 0 1 2 7 0 0 19 12310 VecNorm 38 1.0 9.6422e-01 6.6 1.87e+08 1.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 1 4 0 0 13 4507 VecCopy 38 1.0 4.3037e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 115 1.0 1.0355e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 0 0 0 0 0 VecAXPBYCZ 38 1.0 9.1814e-01 1.1 3.73e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 9466 VecWAXPY 38 1.0 9.1795e-01 1.1 1.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 4 0 0 0 4734 VecAssemblyBegin 38 1.0 8.9392e-0211.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 2 0 0 0 0 38 0 VecAssemblyEnd 38 1.0 4.3392e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 38 1.0 5.8024e-02 2.2 0.00e+00 0.0 1.7e+03 7.2e+05 0.0e+00 0 0 0 2 0 0 0 95 99 0 0 VecScatterEnd 38 1.0 3.1747e-01 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 38 1.0 6.4099e+00 1.1 9.93e+08 1.1 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 17 19 0 0 2 3602 PCSetUpOnBlocks 19 1.0 6.4097e+00 1.1 9.93e+08 1.1 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 17 19 0 0 1 3602 PCApply 57 1.0 6.6614e+00 1.1 1.78e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 18 34 0 0 0 6210 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 6 48 1076255464 0 Krylov Solver 2 9 27392 0 Vector 4 63 248561024 0 Vector Scatter 0 10 10360 0 Index Set 0 10 9824824 0 Preconditioner 0 9 8868 0 Viewer 1 0 0 0 --- Event Stage 1: poisson_eqn Matrix 117 76 734590032 0 Matrix Coarsen 4 4 2448 0 Krylov Solver 10 4 120512 0 Vector 225 175 200939176 0 Vector Scatter 31 22 22792 0 Index Set 81 74 764972 0 Preconditioner 11 4 3424 0 PetscRandom 4 4 2432 0 --- Event Stage 2: momentum_eqn Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1496 0 Vector Scatter 1 0 0 0 Index Set 5 2 361680 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 0 Average time for MPI_Barrier(): 2.36034e-05 Average time for zero size MPI_Send(): 2.29478e-06 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Sun Oct 7 16:51:24 2012 Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --with-mpi-dir=/app1/mvapich2/current/ --with-blas-lapack-dir=/app1/intel/mkl/lib/intel64/ --with-batch=1 --with-debugging=0 --download-hypre=1 --prefix=/home/svu/g0306332/lib/petsc-3.3-dev_atlas6_rel --known-mpi-shared-libraries=0 ----------------------------------------- Libraries compiled on Sun Oct 7 16:51:24 2012 on atlas6-c01 Machine characteristics: Linux-2.6.18-274.7.1.el5-x86_64-with-redhat-5.8-Final Using PETSc directory: /home/svu/g0306332/codes/petsc-dev Using PETSc arch: petsc-3.3-dev_atlas6_rel ----------------------------------------- Using C compiler: /app1/mvapich2/current/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /app1/mvapich2/current/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/app1/mvapich2/current/include -I/app1/mvapich2/1.4/include ----------------------------------------- Using C linker: /app1/mvapich2/current/bin/mpicc Using Fortran linker: /app1/mvapich2/current/bin/mpif90 Using libraries: -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lHYPRE -L/app1/mvapich2/1.4/lib -L/nfs/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/ipp/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichcxx -lstdc++ -lpthread -Wl,-rpath,/app1/intel/mkl/lib/intel64 -L/app1/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpichf90 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl ----------------------------------------- Job /app1/common/lsf7/7.0/linux2.6-glibc2.3-x86_64/bin/mvapich_wrapper ./a.out -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg -log_summary TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME ===== ========== ================ ======================= =================== 00000 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00001 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00002 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00003 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00004 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00005 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00006 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00007 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00008 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00009 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00010 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00011 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:17:35 00012 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00013 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00014 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00015 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00016 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00017 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00018 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00019 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00020 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00021 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00022 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 00023 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:17:35 -------------- next part -------------- ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./a.out on a petsc-3.3-dev_atlas6_rel named atlas6-c14 with 32 processors, by g0306332 Mon Oct 8 04:38:44 2012 Using Petsc Development HG revision: 0b92fc173218fc24fd69e1ee041f811d50d4766c HG Date: Fri Oct 05 15:20:44 2012 -0500 Max Max/Min Avg Total mv: cannot move `/var/tmp/uvwp32.plt' to `/home/wtay/Results/8/': No such file or directory Time (sec): 5.793e+02 1.00264 5.781e+02 Objects: 5.160e+02 1.00000 5.160e+02 Flops: 1.280e+11 1.26041 1.076e+11 3.442e+12 Flops/sec: 2.210e+08 1.25710 1.861e+08 5.955e+09 MPI Messages: 8.329e+04 3.19963 5.696e+04 1.823e+06 MPI Message Lengths: 4.386e+09 2.01368 7.429e+04 1.354e+11 MPI Reductions: 6.648e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.4747e+02 25.5% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 2.000e+01 0.3% 1: poisson_eqn: 4.0301e+02 69.7% 3.3200e+12 96.4% 1.820e+06 99.9% 7.334e+04 98.7% 6.329e+03 95.2% 2: momentum_eqn: 2.7602e+01 4.8% 1.2245e+11 3.6% 2.480e+03 0.1% 9.434e+02 1.3% 2.980e+02 4.5% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage --- Event Stage 1: poisson_eqn MatMult 22328 1.0 2.4333e+02 1.1 8.47e+10 1.3 1.5e+06 8.6e+04 0.0e+00 40 63 80 93 0 57 65 80 94 0 8859 MatMultAdd 3432 1.0 1.7052e+01 1.5 5.96e+09 1.5 1.7e+05 1.8e+04 0.0e+00 2 4 10 2 0 4 4 10 2 0 7952 MatMultTranspose 3432 1.0 2.4190e+01 2.0 5.96e+09 1.5 1.7e+05 1.8e+04 0.0e+00 3 4 10 2 0 4 4 10 2 0 5605 MatSolve 1716 0.0 6.4962e-03 0.0 1.62e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 249 MatLUFactorSym 1 1.0 3.4094e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 1 1.0 2.0027e-05 7.0 6.24e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 311 MatConvert 4 1.0 9.4268e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 12 1.0 1.0746e-01 1.6 2.28e+07 1.4 2.7e+02 8.1e+04 0.0e+00 0 0 0 0 0 0 0 0 0 0 5182 MatAssemblyBegin 69 1.0 2.4791e+00 3.8 0.00e+00 0.0 7.1e+02 1.6e+04 7.4e+01 0 0 0 0 1 0 0 0 0 1 0 MatAssemblyEnd 69 1.0 9.1389e-01 1.1 0.00e+00 0.0 3.8e+03 1.2e+04 2.0e+02 0 0 0 0 3 0 0 0 0 3 0 MatGetRow 1358404 1.1 1.6604e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 0.0 7.1526e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 0.0 4.6015e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.2e-02 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 4 1.0 3.2097e-01 1.3 0.00e+00 0.0 5.8e+03 3.5e+04 2.2e+02 0 0 0 0 3 0 0 0 0 3 0 MatAXPY 4 1.0 2.6504e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 4 1.0 5.9873e-01 1.0 1.59e+07 1.4 1.8e+03 3.8e+04 9.6e+01 0 0 0 0 1 0 0 0 0 2 666 MatMatMultSym 4 1.0 4.7245e-01 1.0 0.00e+00 0.0 1.5e+03 3.1e+04 8.8e+01 0 0 0 0 1 0 0 0 0 1 0 MatMatMultNum 4 1.0 1.2957e-01 1.0 1.59e+07 1.4 2.7e+02 8.1e+04 8.0e+00 0 0 0 0 0 0 0 0 0 0 3078 MatPtAP 4 1.0 2.4452e+00 1.0 6.76e+08 3.0 3.4e+03 7.9e+04 1.1e+02 0 0 0 0 2 1 0 0 0 2 3527 MatPtAPSymbolic 4 1.0 1.5169e+00 1.1 0.00e+00 0.0 3.1e+03 6.5e+04 1.0e+02 0 0 0 0 2 0 0 0 0 2 0 MatPtAPNumeric 4 1.0 9.2885e-01 1.0 6.76e+08 3.0 3.4e+02 2.0e+05 8.0e+00 0 0 0 0 0 0 0 0 0 0 9285 MatTrnMatMult 4 1.0 5.3296e+00 1.0 7.66e+08 3.9 1.5e+03 5.5e+05 1.2e+02 1 1 0 1 2 1 1 0 1 2 4241 MatGetLocalMat 20 1.0 2.7215e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetBrAoCol 12 1.0 9.9058e-02 2.2 0.00e+00 0.0 1.9e+03 1.2e+05 1.6e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetSymTrans 8 1.0 3.3771e-02 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPGMRESOrthog 898 1.0 4.8392e-01 1.4 1.50e+08 1.1 0.0e+00 0.0e+00 9.0e+02 0 0 0 0 14 0 0 0 0 14 9516 KSPSetUp 11 1.0 5.3839e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 20 1.0 4.0076e+02 1.0 1.24e+11 1.3 1.8e+06 7.3e+04 6.2e+03 69 96100 99 93 99100100100 98 8284 VecDot 838 1.0 1.9926e+0110.1 1.02e+09 1.1 0.0e+00 0.0e+00 8.4e+02 1 1 0 0 13 2 1 0 0 13 1603 VecDotNorm2 419 1.0 1.9897e+01 8.4 2.03e+09 1.1 0.0e+00 0.0e+00 1.3e+03 1 2 0 0 19 2 2 0 0 20 3211 VecMDot 898 1.0 3.8118e-01 1.9 7.48e+07 1.1 0.0e+00 0.0e+00 9.0e+02 0 0 0 0 14 0 0 0 0 14 6040 VecNorm 2199 1.0 1.4204e+0118.3 5.47e+08 1.1 0.0e+00 0.0e+00 2.2e+03 1 0 0 0 33 1 1 0 0 35 1211 VecScale 15488 1.0 9.1676e+00 2.1 2.34e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 2 2 0 0 0 7861 VecCopy 4334 1.0 3.1940e+00 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 14721 1.0 2.4871e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 28318 1.0 3.1641e+01 1.9 9.33e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 8 0 0 0 7 9 0 0 0 9083 VecAYPX 27456 1.0 3.1435e+01 1.6 5.83e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 5 0 0 0 7 5 0 0 0 5713 VecAXPBYCZ 838 1.0 4.9286e+00 1.3 2.03e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 12962 VecWAXPY 838 1.0 4.9716e+00 1.3 1.02e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 6425 VecMAXPY 1760 1.0 1.7820e-01 1.5 8.84e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 15270 VecAssemblyBegin 118 1.0 1.9584e+00 6.7 0.00e+00 0.0 0.0e+00 0.0e+00 3.5e+02 0 0 0 0 5 0 0 0 0 5 0 VecAssemblyEnd 118 1.0 1.0204e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 20636 1.0 3.3645e+01 1.7 3.50e+09 1.1 0.0e+00 0.0e+00 0.0e+00 5 3 0 0 0 7 3 0 0 0 3210 VecScatterBegin 29278 1.0 4.1807e+00 2.5 0.00e+00 0.0 1.8e+06 7.3e+04 0.0e+00 1 0 99 98 0 1 0100 99 0 0 VecScatterEnd 29278 1.0 1.0144e+0214.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 8 0 0 0 0 0 VecSetRandom 4 1.0 1.0141e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 1760 1.0 1.1541e+01158.2 2.25e+07 1.1 0.0e+00 0.0e+00 1.8e+03 1 0 0 0 26 1 0 0 0 28 60 PCSetUp 2 1.0 1.2079e+01 1.0 1.36e+09 1.2 2.0e+04 9.8e+04 1.1e+03 2 1 1 1 16 3 1 1 1 17 3473 PCSetUpOnBlocks 858 1.0 8.4829e-04 1.7 6.24e+03 0.0 0.0e+00 0.0e+00 5.0e+00 0 0 0 0 0 0 0 0 0 0 7 PCApply 858 1.0 3.4936e+02 1.1 1.11e+11 1.3 1.7e+06 6.8e+04 2.6e+03 59 83 96 88 39 84 86 96 89 41 8195 PCGAMGgraph_AGG 4 1.0 1.1721e+00 1.0 1.59e+07 1.4 7.4e+02 4.4e+04 7.6e+01 0 0 0 0 1 0 0 0 0 1 340 PCGAMGcoarse_AGG 4 1.0 5.8811e+00 1.0 7.66e+08 3.9 8.7e+03 1.4e+05 4.1e+02 1 1 0 1 6 1 1 0 1 7 3843 PCGAMGProl_AGG 4 1.0 8.9037e-01 1.0 0.00e+00 0.0 1.8e+03 5.4e+04 1.1e+02 0 0 0 0 2 0 0 0 0 2 0 PCGAMGPOpt_AGG 4 1.0 1.6679e+00 1.0 3.64e+08 1.2 4.5e+03 6.4e+04 2.1e+02 0 0 0 0 3 0 0 0 0 3 6189 --- Event Stage 2: momentum_eqn MatMult 38 1.0 3.3929e+00 1.3 8.88e+08 1.1 2.4e+03 7.2e+05 0.0e+00 1 1 0 1 0 12 23 95 99 0 8221 MatSolve 57 1.0 4.6989e+00 1.4 1.31e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 16 34 0 0 0 8769 MatLUFactorNum 19 1.0 4.5038e+00 1.1 7.31e+08 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 16 19 0 0 0 5093 MatILUFactorSym 1 1.0 1.9833e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 1 0 0 0 0 0 MatAssemblyBegin 19 1.0 3.3288e+0018.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 5 0 0 0 13 0 MatAssemblyEnd 19 1.0 1.6178e+00 1.2 0.00e+00 0.0 1.2e+02 1.8e+05 8.0e+00 0 0 0 0 0 6 0 5 1 3 0 MatGetRowIJ 1 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.2255e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 1 0 KSPSetUp 38 1.0 5.9859e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 19 1.0 1.6160e+01 1.0 3.90e+09 1.1 2.4e+03 7.2e+05 1.4e+02 3 4 0 1 2 59100 95 99 46 7577 VecDot 38 1.0 1.3321e+00 4.0 1.38e+08 1.1 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 2 4 0 0 13 3262 VecDotNorm2 19 1.0 1.2704e+00 3.6 2.76e+08 1.1 0.0e+00 0.0e+00 5.7e+01 0 0 0 0 1 2 7 0 0 19 6841 VecNorm 38 1.0 1.5091e+00 9.8 1.38e+08 1.1 0.0e+00 0.0e+00 3.8e+01 0 0 0 0 1 2 4 0 0 13 2879 VecCopy 38 1.0 3.3406e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 115 1.0 7.8406e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 0 0 0 0 0 VecAXPBYCZ 38 1.0 6.9826e-01 1.4 2.76e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 7 0 0 0 12446 VecWAXPY 38 1.0 6.8325e-01 1.3 1.38e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 4 0 0 0 6360 VecAssemblyBegin 38 1.0 4.2442e-0143.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 2 0 0 0 0 38 0 VecAssemblyEnd 38 1.0 4.1008e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 38 1.0 5.7228e-02 2.5 0.00e+00 0.0 2.4e+03 7.2e+05 0.0e+00 0 0 0 1 0 0 0 95 99 0 0 VecScatterEnd 38 1.0 4.9354e-01 4.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 PCSetUp 38 1.0 4.7225e+00 1.1 7.31e+08 1.1 0.0e+00 0.0e+00 5.0e+00 1 1 0 0 0 16 19 0 0 2 4857 PCSetUpOnBlocks 19 1.0 4.7223e+00 1.1 7.31e+08 1.1 0.0e+00 0.0e+00 3.0e+00 1 1 0 0 0 16 19 0 0 1 4857 PCApply 57 1.0 5.0658e+00 1.4 1.31e+09 1.1 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 17 34 0 0 0 8133 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 6 48 802816820 0 Krylov Solver 2 9 27392 0 Vector 4 63 184694312 0 Vector Scatter 0 10 10360 0 Index Set 0 10 7279624 0 Preconditioner 0 9 8868 0 Viewer 1 0 0 0 --- Event Stage 1: poisson_eqn Matrix 117 76 536993068 0 Matrix Coarsen 4 4 2448 0 Krylov Solver 10 4 120512 0 Vector 225 175 150834928 0 Vector Scatter 31 22 22792 0 Index Set 81 74 749444 0 Preconditioner 11 4 3424 0 PetscRandom 4 4 2432 0 --- Event Stage 2: momentum_eqn Matrix 1 0 0 0 Krylov Solver 1 0 0 0 Vector 10 1 1496 0 Vector Scatter 1 0 0 0 Index Set 5 2 361680 0 Preconditioner 2 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 9.53674e-08 Average time for MPI_Barrier(): 2.57969e-05 Average time for zero size MPI_Send(): 2.34693e-06 #PETSc Option Table entries: -log_summary -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Sun Oct 7 16:51:24 2012 Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --with-mpi-dir=/app1/mvapich2/current/ --with-blas-lapack-dir=/app1/intel/mkl/lib/intel64/ --with-batch=1 --with-debugging=0 --download-hypre=1 --prefix=/home/svu/g0306332/lib/petsc-3.3-dev_atlas6_rel --known-mpi-shared-libraries=0 ----------------------------------------- Libraries compiled on Sun Oct 7 16:51:24 2012 on atlas6-c01 Machine characteristics: Linux-2.6.18-274.7.1.el5-x86_64-with-redhat-5.8-Final Using PETSc directory: /home/svu/g0306332/codes/petsc-dev Using PETSc arch: petsc-3.3-dev_atlas6_rel ----------------------------------------- Using C compiler: /app1/mvapich2/current/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /app1/mvapich2/current/bin/mpif90 -O3 ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/include -I/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/include -I/app1/mvapich2/current/include -I/app1/mvapich2/1.4/include ----------------------------------------- Using C linker: /app1/mvapich2/current/bin/mpicc Using Fortran linker: /app1/mvapich2/current/bin/mpif90 Using libraries: -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lpetsc -lX11 -Wl,-rpath,/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -L/home/svu/g0306332/codes/petsc-dev/petsc-3.3-dev_atlas6_rel/lib -lHYPRE -L/app1/mvapich2/1.4/lib -L/nfs/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/ipp/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -L/app1/intel_2011/composer_xe_2011_sp1.6.233/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichcxx -lstdc++ -lpthread -Wl,-rpath,/app1/intel/mkl/lib/intel64 -L/app1/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lmpichf90 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -lmpichcxx -lstdc++ -lmpichcxx -lstdc++ -ldl -lmpich -lpthread -lrdmacm -libverbs -libumad -lrt -lgcc_s -ldl ----------------------------------------- Job /app1/common/lsf7/7.0/linux2.6-glibc2.3-x86_64/bin/mvapich_wrapper ./a.out -poisson_pc_gamg_agg_nsmooths 1 -poisson_pc_type gamg -log_summary TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME ===== ========== ================ ======================= =================== 00000 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00001 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00002 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00003 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00004 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00005 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00006 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00007 atlas6-c43 ./a.out -poisson Done 10/08/2012 04:38:45 00008 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00009 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00010 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00011 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00012 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00013 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00014 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00015 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00016 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00017 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00018 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00019 atlas6-c39 ./a.out -poisson Done 10/08/2012 04:38:45 00020 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00021 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00022 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00023 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00024 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00025 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00026 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00027 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00028 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00029 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00030 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 00031 atlas6-c14 ./a.out -poisson Done 10/08/2012 04:38:45 From zonexo at gmail.com Mon Oct 8 09:19:04 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 08 Oct 2012 16:19:04 +0200 Subject: [petsc-users] Error with PETSc logging Message-ID: <5072E0D8.5040303@gmail.com> Hi, I am trying to log my PETSc performance in stages. I initially used: /PetscLogStage stages(2)// // //do time = 1,n// //// // ...// // // call PetscLogStageRegister("momentum_eqn",stages(1),ierr)// // // call PetscLogStagePush(stages(1),ierr)// // // stage 1 logging// // // call PetscLogStagePop(ierr)// // // ...// // // call PetscLogStageRegister("poisson_eqn",stages(2),ierr)// // // call PetscLogStagePush(stages(2),ierr)// // // stage 2 logging// // // call PetscLogStagePop(ierr)// // //end do/ The log_summary shows that results in stages 1 and 2, but I also get the error: Invalid argument! [1]PETSC ERROR: Duplicate stage name given: poisson_eqn! I changed my code to only call PetscLogStageRegister once: /PetscLogStage stages(2)// // //call PetscLogStageRegister("momentum_eqn",stages(1),ierr)// // //call PetscLogStageRegister("poisson_eqn",stages(2),ierr)// / / //do time = 1,n// //// // ...// / / // call PetscLogStagePush(stages(1),ierr)// / / // stage 1 logging// / / // call PetscLogStagePop(ierr)// / / ////...// / / // call PetscLogStagePush(stages(2),ierr)// / / // stage 2 logging// / / // call PetscLogStagePop(ierr)// / / //end do/ Now there's no error but all logging are now grouped together into just 1 main stage. How can I correct it to have separate logging? Tks! -- Yours sincerely, TAY wee-beng -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Mon Oct 8 13:37:18 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Oct 2012 13:37:18 -0500 Subject: [petsc-users] Enquiry regarding log summary results In-Reply-To: <5071F7E0.5090103@gmail.com> References: <5064051D.1060903@gmail.com> <506C38DE.7030107@gmail.com> <506C9A99.60300@gmail.com> <506DA4E0.5040503@gmail.com> <506E01CF.8080109@gmail.com> <506EF7F3.3040709@gmail.com> <5070A07B.3080900@gmail.com> <5070A7EE.4060408@gmail.com> <5071F7E0.5090103@gmail.com> Message-ID: On Sun, Oct 7, 2012 at 4:45 PM, TAY wee-beng wrote: > Hi, > > I have attached 3 results using 12,24 and 32 processors. I am using a > completely different clusters and I'm testing if it's the cluster > configuration problems. It's seems that VecScatterEnd does not scale well > from 12 to 24 to 32. Does these results show that there's problems with the > partition still? My partition is clustered closely at the center, I'm > wondering if this has a great effect on scaling... > Your partition is probably not well balanced, but you're also seeing what happens when an algorithm is limited by bandwidth. The vendors tend not to tell you that cores don't matter unless you can get memory to them. Sparse linear algebra is heavily bandwidth-limited so you should not expect to see speedup by using all cores on a node. If you use additional nodes, you will get additional memory buses which are a better indicator of performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Mon Oct 8 14:04:12 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Mon, 8 Oct 2012 15:04:12 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <5071B4E8.6050003@gmail.com> References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> <5071B4E8.6050003@gmail.com> Message-ID: <67C5860A-6F0D-42A3-A48C-31C1F4146DCC@columbia.edu> Alan, I have recently put PETSc in Chombo. You seem to be thinking about at putting something like Chombo in PETSc. Do you have an AMR code? The BoxLib project from LBNL also has a PETSc interface. I did not follow this whole thread but, and this was probably already stated, PETSc does not have an interface to HYPRE's block structured solvers (I don't think). Also, HYPRE does not really do AMR; they do multi-block solvers, which amounts to much larger blocks. Its not clear to me how appropriate their current solvers are for AMR. Do you know? That said, HYPRE is actively working with BoxLib on solvers and so I would expect HYPRE to have more AMR solver functionality in the future, but that will not be available in PETSc for a long time (if ever). Mark On Oct 7, 2012, at 12:59 PM, Alan wrote: > Dear Dr. Brown, > I was trying to use PETSc to mimic something like Chombo. The figure in this website represents my thought very well. (https://commons.lbl.gov/display/chombo/Chombo+-+Software+for+Adaptive+Solutions+of+Partial+Differential+Equations). My code simulates a flow over a objective, i.e. 2D cylinder. Finer resolution is needed near the objective. I can manually define where the finer resolution is needed. > BTW, I wonder if there is any examples for the Vec/Mat interface. Also, is there any examples for PETSc for GPU. > > thanks, > Alan >> On Sat, Oct 6, 2012 at 10:47 PM, Alan wrote: >> Dear Dr. Brown, >> Thanks for your reply and sorry for confusing you. >> I was trying to ask if there is an alternative way that I can replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with something from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a 3D Cartesian grid with uniform grid size. However, I'm trying to import a local refined grid from outside of the PETSc (i.e. HYPRE) to replace this 3D Cartesian grid with unifrom grid size. Is there any examples for me to refer? >> >> Do we have automatic support for this? No. You could of course just use the Vec/Mat interface, but >> it sounds like a lot of coding on your part. I plan to do this eventually by specifying the data layout over >> the grid using PetscSection. However, the really important question here is, what do you need from the >> grid? If you don't need topological queries, just make the Vec/Mat and be done with it. If you do (which >> I imagine), then you need to be specific about what you need. >> >> Matt >> >> thanks, >> Alan >> >>> Run any example with -pc_type hypre. I'm not sure what you're asking for. >>> >>> On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: >>> Dear Folks, >>> I hope you're having a nice day. >>> I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to >>> obtain solutions (Poisson Equation) with local refined grid from HYPRE. >>> Is there any examples in PETSc that I can refer? or is there any >>> examples in PETSc which introduce such a way that the Matrix is imported >>> from external software rather than generated by PETSc. >>> >>> thanks, >>> Alan >>> >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Mon Oct 8 15:16:20 2012 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Mon, 08 Oct 2012 16:16:20 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <67C5860A-6F0D-42A3-A48C-31C1F4146DCC@columbia.edu> References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> <5071B4E8.6050003@gmail.com> <67C5860A-6F0D-42A3-A48C-31C1F4146DCC@columbia.edu> Message-ID: <50733494.3040704@cims.nyu.edu> On 10/8/12 3:04 PM, Mark F. Adams wrote: > I did not follow this whole thread but, and this was probably already > stated, PETSc does not have an interface to HYPRE's block structured > solvers (I don't think). Also, HYPRE does not really do AMR; they do > multi-block solvers, which amounts to much larger blocks. Its not clear > to me how appropriate their current solvers are for AMR. Do you know? hypre provides a FAC preconditioner (or at least used to provide one --- I haven't checked recently) through the SStruct interface. I could never quite figure out how the FAC preconditioner interface worked, though, and was never to get anything running with it. YMMV. -- Boyce From jedbrown at mcs.anl.gov Mon Oct 8 13:11:42 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Mon, 8 Oct 2012 13:11:42 -0500 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <5071B4E8.6050003@gmail.com> References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> <5071B4E8.6050003@gmail.com> Message-ID: On Sun, Oct 7, 2012 at 11:59 AM, Alan wrote: > Dear Dr. Brown, > I was trying to use PETSc to mimic something like Chombo. The figure > in this website represents my thought very well. ( > https://commons.lbl.gov/display/chombo/Chombo+-+Software+for+Adaptive+Solutions+of+Partial+Differential+Equations). > My code simulates a flow over a objective, i.e. 2D cylinder. Finer > resolution is needed near the objective. I can manually define where the > finer resolution is needed. > You could just use Chombo... It's a lot of work to write a good implementation of dynamic AMR. We don't have a fully worked example of doing that, but there are plenty of packages that do AMR and use PETSc. > BTW, I wonder if there is any examples for the Vec/Mat interface. > See src/mat/examples/tutorials, or any of the higher level interfaces. Look at the user's manual. > Also, is there any examples for PETSc for GPU. > http://www.mcs.anl.gov/petsc/features/gpus.html > > thanks, > Alan > > On Sat, Oct 6, 2012 at 10:47 PM, Alan wrote: > >> Dear Dr. Brown, >> Thanks for your reply and sorry for confusing you. >> I was trying to ask if there is an alternative way that I can >> replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with >> something from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a >> 3D Cartesian grid with uniform grid size. However, I'm trying to import a >> local refined grid from outside of the PETSc (i.e. HYPRE) to replace this >> 3D Cartesian grid with unifrom grid size. Is there any examples for me to >> refer? >> > > Do we have automatic support for this? No. You could of course just use > the Vec/Mat interface, but > it sounds like a lot of coding on your part. I plan to do this eventually > by specifying the data layout over > the grid using PetscSection. However, the really important question here > is, what do you need from the > grid? If you don't need topological queries, just make the Vec/Mat and be > done with it. If you do (which > I imagine), then you need to be specific about what you need. > > Matt > > >> thanks, >> Alan >> >> Run any example with -pc_type hypre. I'm not sure what you're asking >> for. >> >> On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: >> >>> Dear Folks, >>> I hope you're having a nice day. >>> I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to >>> obtain solutions (Poisson Equation) with local refined grid from HYPRE. >>> Is there any examples in PETSc that I can refer? or is there any >>> examples in PETSc which introduce such a way that the Matrix is imported >>> from external software rather than generated by PETSc. >>> >>> thanks, >>> Alan >>> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Oct 8 16:39:02 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Oct 2012 17:39:02 -0400 Subject: [petsc-users] Error with PETSc logging In-Reply-To: <5072E0D8.5040303@gmail.com> References: <5072E0D8.5040303@gmail.com> Message-ID: Register the stages outside the time loop. Matt On Mon, Oct 8, 2012 at 10:19 AM, TAY wee-beng wrote: > Hi, > > I am trying to log my PETSc performance in stages. > > I initially used: > > *PetscLogStage stages(2)** > ** > **do time = 1,n** > ** ** > ** ...** > ** > ** call PetscLogStageRegister("momentum_eqn",stages(1),ierr)** > ** > ** call PetscLogStagePush(stages(1),ierr)** > ** > ** stage 1 logging** > ** > ** call PetscLogStagePop(ierr)** > ** > ** ...** > ** > ** call PetscLogStageRegister("poisson_eqn",stages(2),ierr)** > ** > ** call PetscLogStagePush(stages(2),ierr)** > ** > ** stage 2 logging** > ** > ** call PetscLogStagePop(ierr)** > ** > **end do* > > > The log_summary shows that results in stages 1 and 2, but I also get the > error: > > Invalid argument! > [1]PETSC ERROR: Duplicate stage name given: poisson_eqn! > > I changed my code to only call PetscLogStageRegister once: > > *PetscLogStage stages(2)** > ** > **call PetscLogStageRegister("momentum_eqn",stages(1),ierr)** > ** > ** call PetscLogStageRegister("poisson_eqn",stages(2),ierr)** > * * > ** do time = 1,n** > ** ** > ** ...** > * * > ** call PetscLogStagePush(stages(1),ierr)** > * * > ** stage 1 logging** > * * > ** call PetscLogStagePop(ierr)** > * * > ** **...** > * * > ** call PetscLogStagePush(stages(2),ierr)** > * * > ** stage 2 logging** > * * > ** call PetscLogStagePop(ierr)** > * * > ** end do* > > Now there's no error but all logging are now grouped together into just 1 > main stage. > > How can I correct it to have separate logging? Tks! > > -- > Yours sincerely, > > TAY wee-beng > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Mon Oct 8 16:44:44 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Mon, 08 Oct 2012 23:44:44 +0200 Subject: [petsc-users] Error with PETSc logging In-Reply-To: References: <5072E0D8.5040303@gmail.com> Message-ID: <5073494C.3070509@gmail.com> On 8/10/2012 11:39 PM, Matthew Knepley wrote: > Register the stages outside the time loop. > > Matt Hi Matt, I did that in my 2nd attempt (as in the org email), but it only shows 1 main stage in the end: I changed my code to only call PetscLogStageRegister once: /PetscLogStage stages(2)// // //call PetscLogStageRegister("momentum_eqn",stages(1),ierr)// // //call PetscLogStageRegister("poisson_eqn",stages(2),ierr)// / / //do time = 1,n// //// // ...// / / // call PetscLogStagePush(stages(1),ierr)// / / // stage 1 logging// / / // call PetscLogStagePop(ierr)// / / ////...// / / // call PetscLogStagePush(stages(2),ierr)// / / // stage 2 logging// / / // call PetscLogStagePop(ierr)// / / //end do/ > > On Mon, Oct 8, 2012 at 10:19 AM, TAY wee-beng > wrote: > > Hi, > > I am trying to log my PETSc performance in stages. > > I initially used: > > /PetscLogStage stages(2)// > // > //do time = 1,n// > //// > // ...// > // > // call PetscLogStageRegister("momentum_eqn",stages(1),ierr)// > // > // call PetscLogStagePush(stages(1),ierr)// > // > // stage 1 logging// > // > // call PetscLogStagePop(ierr)// > // > // ...// > // > // call PetscLogStageRegister("poisson_eqn",stages(2),ierr)// > // > // call PetscLogStagePush(stages(2),ierr)// > // > // stage 2 logging// > // > // call PetscLogStagePop(ierr)// > // > //end do/ > > > The log_summary shows that results in stages 1 and 2, but I also > get the error: > > Invalid argument! > [1]PETSC ERROR: Duplicate stage name given: poisson_eqn! > > I changed my code to only call PetscLogStageRegister once: > > /PetscLogStage stages(2)// > // > //call PetscLogStageRegister("momentum_eqn",stages(1),ierr)// > // > //call PetscLogStageRegister("poisson_eqn",stages(2),ierr)// > / / > //do time = 1,n// > //// > // ...// > / / > // call PetscLogStagePush(stages(1),ierr)// > / / > // stage 1 logging// > / / > // call PetscLogStagePop(ierr)// > / / > ////...// > / / > // call PetscLogStagePush(stages(2),ierr)// > / / > // stage 2 logging// > / / > // call PetscLogStagePop(ierr)// > / / > //end do/ > > Now there's no error but all logging are now grouped together into > just 1 main stage. > > How can I correct it to have separate logging? Tks! > > -- > Yours sincerely, > > TAY wee-beng > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Mon Oct 8 16:52:09 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Mon, 8 Oct 2012 17:52:09 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: References: <5070D539.905@gmail.com> <5070ED55.8060709@gmail.com> <5071B4E8.6050003@gmail.com> Message-ID: On Oct 8, 2012, at 2:11 PM, Jed Brown wrote: > On Sun, Oct 7, 2012 at 11:59 AM, Alan wrote: > Dear Dr. Brown, > I was trying to use PETSc to mimic something like Chombo. The figure in this website represents my thought very well. (https://commons.lbl.gov/display/chombo/Chombo+-+Software+for+Adaptive+Solutions+of+Partial+Differential+Equations). My code simulates a flow over a objective, i.e. 2D cylinder. Finer resolution is needed near the objective. I can manually define where the finer resolution is needed. > > You could just use Chombo... It's a lot of work to write a good implementation of dynamic AMR. We don't have a fully worked example of doing that, but there are plenty of packages that do AMR and use PETSc. You should also consider BoxLib, which comes from the same hallway at LBNL and both projects are lead by a member of the NAS. BoxLib is less C++ and it has a more sophisticated interface to PETSc (better than mine from what I know). Chombo has support for multi block grids and embedded boundaries. I've worked a lot with Chombo and it is a nice product but there is definitely a learning curve, especially if you are not experienced with C++. I've not worked with BoxLib but I'm sure it is good also. Mark > > BTW, I wonder if there is any examples for the Vec/Mat interface. > > See src/mat/examples/tutorials, or any of the higher level interfaces. Look at the user's manual. > > Also, is there any examples for PETSc for GPU. > > http://www.mcs.anl.gov/petsc/features/gpus.html > > > thanks, > Alan >> On Sat, Oct 6, 2012 at 10:47 PM, Alan wrote: >> Dear Dr. Brown, >> Thanks for your reply and sorry for confusing you. >> I was trying to ask if there is an alternative way that I can replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with something from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a 3D Cartesian grid with uniform grid size. However, I'm trying to import a local refined grid from outside of the PETSc (i.e. HYPRE) to replace this 3D Cartesian grid with unifrom grid size. Is there any examples for me to refer? >> >> Do we have automatic support for this? No. You could of course just use the Vec/Mat interface, but >> it sounds like a lot of coding on your part. I plan to do this eventually by specifying the data layout over >> the grid using PetscSection. However, the really important question here is, what do you need from the >> grid? If you don't need topological queries, just make the Vec/Mat and be done with it. If you do (which >> I imagine), then you need to be specific about what you need. >> >> Matt >> >> thanks, >> Alan >> >>> Run any example with -pc_type hypre. I'm not sure what you're asking for. >>> >>> On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: >>> Dear Folks, >>> I hope you're having a nice day. >>> I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to >>> obtain solutions (Poisson Equation) with local refined grid from HYPRE. >>> Is there any examples in PETSc that I can refer? or is there any >>> examples in PETSc which introduce such a way that the Matrix is imported >>> from external software rather than generated by PETSc. >>> >>> thanks, >>> Alan >>> >>> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Oct 8 17:19:21 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Oct 2012 18:19:21 -0400 Subject: [petsc-users] Error with PETSc logging In-Reply-To: <5073494C.3070509@gmail.com> References: <5072E0D8.5040303@gmail.com> <5073494C.3070509@gmail.com> Message-ID: Then it was done wrong. There are lots of examples that do this Matt On Oct 8, 2012 4:44 PM, "TAY wee-beng" wrote: > > On 8/10/2012 11:39 PM, Matthew Knepley wrote: > > Register the stages outside the time loop. > > Matt > > > Hi Matt, > > I did that in my 2nd attempt (as in the org email), but it only shows 1 > main stage in the end: > > I changed my code to only call PetscLogStageRegister once: > > *PetscLogStage stages(2)** > ** > **call PetscLogStageRegister("momentum_eqn",stages(1),ierr)** > ** > ** call PetscLogStageRegister("poisson_eqn",stages(2),ierr)** > * * > ** do time = 1,n** > ** ** > ** ...** > * * > ** call PetscLogStagePush(stages(1),ierr)** > * * > ** stage 1 logging** > * * > ** call PetscLogStagePop(ierr)** > * * > ** **...** > * * > ** call PetscLogStagePush(stages(2),ierr)** > * * > ** stage 2 logging** > * * > ** call PetscLogStagePop(ierr)** > * * > ** end do* > > > On Mon, Oct 8, 2012 at 10:19 AM, TAY wee-beng wrote: > >> Hi, >> >> I am trying to log my PETSc performance in stages. >> >> I initially used: >> >> *PetscLogStage stages(2)** >> ** >> **do time = 1,n** >> ** ** >> ** ...** >> ** >> ** call PetscLogStageRegister("momentum_eqn",stages(1),ierr)** >> ** >> ** call PetscLogStagePush(stages(1),ierr)** >> ** >> ** stage 1 logging** >> ** >> ** call PetscLogStagePop(ierr)** >> ** >> ** ...** >> ** >> ** call PetscLogStageRegister("poisson_eqn",stages(2),ierr)** >> ** >> ** call PetscLogStagePush(stages(2),ierr)** >> ** >> ** stage 2 logging** >> ** >> ** call PetscLogStagePop(ierr)** >> ** >> **end do* >> >> >> The log_summary shows that results in stages 1 and 2, but I also get the >> error: >> >> Invalid argument! >> [1]PETSC ERROR: Duplicate stage name given: poisson_eqn! >> >> I changed my code to only call PetscLogStageRegister once: >> >> *PetscLogStage stages(2)** >> ** >> **call PetscLogStageRegister("momentum_eqn",stages(1),ierr)** >> ** >> ** call PetscLogStageRegister("poisson_eqn",stages(2),ierr)** >> * * >> ** do time = 1,n** >> ** ** >> ** ...** >> * * >> ** call PetscLogStagePush(stages(1),ierr)** >> * * >> ** stage 1 logging** >> * * >> ** call PetscLogStagePop(ierr)** >> * * >> ** **...** >> * * >> ** call PetscLogStagePush(stages(2),ierr)** >> * * >> ** stage 2 logging** >> * * >> ** call PetscLogStagePop(ierr)** >> * * >> ** end do* >> >> Now there's no error but all logging are now grouped together into just 1 >> main stage. >> >> How can I correct it to have separate logging? Tks! >> >> -- >> Yours sincerely, >> >> TAY wee-beng >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Tue Oct 9 08:48:50 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Tue, 9 Oct 2012 09:48:50 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <50739DD1.2090200@gmail.com> References: <50739DD1.2090200@gmail.com> Message-ID: <32EDF70A-37EC-4D01-AE27-07931B45A95F@columbia.edu> AMR is the kind of thing that you put your physics into as opposed to putting AMR in your code. As I said before Chombo and BoxLib are two good options (and probably your only real options). Flow is a common application and both libraries should work for you. On Oct 8, 2012, at 11:45 PM, Alan wrote: > Dear Dr. Adams, > Thank you so much for your knowledgable explanation. > You're right. I do want to implement something like Chombo in PETSc, since I wish my program can solve a flow problem with a local refined Cartisian mesh. However, it seems to be impossible for me to do so. My program is not designed for AMR code; however, I can change its structure for AMR as long as I have a Poisson Solver AMR method. Do you have any suggestions on this? :) > > thanks, > Alan > >> Alan, I have recently put PETSc in Chombo. You seem to be thinking about at putting something like Chombo in PETSc. Do you have an AMR code? >> >> The BoxLib project from LBNL also has a PETSc interface. >> >> I did not follow this whole thread but, and this was probably already stated, PETSc does not have an interface to HYPRE's block structured solvers (I don't think). Also, HYPRE does not really do AMR; they do multi-block solvers, which amounts to much larger blocks. Its not clear to me how appropriate their current solvers are for AMR. Do you know? >> >> That said, HYPRE is actively working with BoxLib on solvers and so I would expect HYPRE to have more AMR solver functionality in the future, but that will not be available in PETSc for a long time (if ever). >> >> Mark >> >> On Oct 7, 2012, at 12:59 PM, Alan wrote: >> >>> Dear Dr. Brown, >>> I was trying to use PETSc to mimic something like Chombo. The figure in this website represents my thought very well. (https://commons.lbl.gov/display/chombo/Chombo+-+Software+for+Adaptive+Solutions+of+Partial+Differential+Equations). My code simulates a flow over a objective, i.e. 2D cylinder. Finer resolution is needed near the objective. I can manually define where the finer resolution is needed. >>> BTW, I wonder if there is any examples for the Vec/Mat interface. Also, is there any examples for PETSc for GPU. >>> >>> thanks, >>> Alan >>>> On Sat, Oct 6, 2012 at 10:47 PM, Alan wrote: >>>> Dear Dr. Brown, >>>> Thanks for your reply and sorry for confusing you. >>>> I was trying to ask if there is an alternative way that I can replace the DMDACreate3d in /src/ksp/ksp/example/tutorials/ex45.c with something from outside of the PETSc (i.e. HYPRE). DMDACreate3d generates a 3D Cartesian grid with uniform grid size. However, I'm trying to import a local refined grid from outside of the PETSc (i.e. HYPRE) to replace this 3D Cartesian grid with unifrom grid size. Is there any examples for me to refer? >>>> >>>> Do we have automatic support for this? No. You could of course just use the Vec/Mat interface, but >>>> it sounds like a lot of coding on your part. I plan to do this eventually by specifying the data layout over >>>> the grid using PetscSection. However, the really important question here is, what do you need from the >>>> grid? If you don't need topological queries, just make the Vec/Mat and be done with it. If you do (which >>>> I imagine), then you need to be specific about what you need. >>>> >>>> Matt >>>> >>>> thanks, >>>> Alan >>>> >>>>> Run any example with -pc_type hypre. I'm not sure what you're asking for. >>>>> >>>>> On Sat, Oct 6, 2012 at 8:04 PM, Alan wrote: >>>>> Dear Folks, >>>>> I hope you're having a nice day. >>>>> I'm trying to couple HYPRE with PETSc in order to use PETSc solvers to >>>>> obtain solutions (Poisson Equation) with local refined grid from HYPRE. >>>>> Is there any examples in PETSc that I can refer? or is there any >>>>> examples in PETSc which introduce such a way that the Matrix is imported >>>>> from external software rather than generated by PETSc. >>>>> >>>>> thanks, >>>>> Alan >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From griffith at cims.nyu.edu Tue Oct 9 09:25:13 2012 From: griffith at cims.nyu.edu (Boyce Griffith) Date: Tue, 09 Oct 2012 10:25:13 -0400 Subject: [petsc-users] Coupling HYPRE with PETSc for local grid refinement In-Reply-To: <32EDF70A-37EC-4D01-AE27-07931B45A95F@columbia.edu> References: <50739DD1.2090200@gmail.com> <32EDF70A-37EC-4D01-AE27-07931B45A95F@columbia.edu> Message-ID: <507433C9.2010404@cims.nyu.edu> On 10/9/12 9:48 AM, Mark F. Adams wrote: > AMR is the kind of thing that you put your physics into as opposed to > putting AMR in your code. As I said before Chombo and BoxLib are two > good options (and probably your only real options). Flow is a common > application and both libraries should work for you. There is also SAMRAI: https://computation.llnl.gov/casc/SAMRAI/index.html which, despite the lack of updates to the website, is still being actively developed and released. -- Boyce From thomas.witkowski at tu-dresden.de Wed Oct 10 08:30:27 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 10 Oct 2012 15:30:27 +0200 Subject: [petsc-users] Convert MatNest to MatAIJ Message-ID: <50757873.7010704@tu-dresden.de> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use MatConvert, but it seems that this convert is not yet implemented. As I need this functionality to factorize the nested matrix with MUMPS, is there any better way than to extract the nested matrices, iterate over all rows of these matrices and use MatSetValues to add them again to the final MatAIJ? Thomas From jedbrown at mcs.anl.gov Wed Oct 10 08:33:28 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 08:33:28 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: <50757873.7010704@tu-dresden.de> References: <50757873.7010704@tu-dresden.de> Message-ID: The best way, by far, is to assemble the matrix as AIJ if that is what you want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST should he a run-time choice. On Oct 10, 2012 8:30 AM, "Thomas Witkowski" wrote: > Is there a fast way to convert a MatNest to a MatAIJ? I tried to use > MatConvert, but it seems that this convert is not yet implemented. As I > need this functionality to factorize the nested matrix with MUMPS, is there > any better way than to extract the nested matrices, iterate over all rows > of these matrices and use MatSetValues to add them again to the final > MatAIJ? > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 10 08:45:41 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 10 Oct 2012 15:45:41 +0200 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> Message-ID: <50757C05.20406@tu-dresden.de> You are right. But what if a have a MATNEST and i must convert it to MATAIJ. Any better way than what I wrote in my first mail? Thomas Am 10.10.2012 15:33, schrieb Jed Brown: > > The best way, by far, is to assemble the matrix as AIJ if that is what > you want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST > should he a run-time choice. > > On Oct 10, 2012 8:30 AM, "Thomas Witkowski" > > wrote: > > Is there a fast way to convert a MatNest to a MatAIJ? I tried to > use MatConvert, but it seems that this convert is not yet > implemented. As I need this functionality to factorize the nested > matrix with MUMPS, is there any better way than to extract the > nested matrices, iterate over all rows of these matrices and use > MatSetValues to add them again to the final MatAIJ? > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Oct 10 08:48:42 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Oct 2012 09:48:42 -0400 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: <50757C05.20406@tu-dresden.de> References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> Message-ID: On Wed, Oct 10, 2012 at 9:45 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > You are right. But what if a have a MATNEST and i must convert it to > MATAIJ. Any better way than what I wrote in my first mail? > I think you miss Jed's point. You should NEVER have a MATNEST. You should have a Mat. Its type should be determined by MatSetFromOptions(). If you want MATAIJ, you give -mat_type aij, and if you want MATNEST, you give -mat_type nest. Thanks, Matt > Thomas > > Am 10.10.2012 15:33, schrieb Jed Brown: > > The best way, by far, is to assemble the matrix as AIJ if that is what you > want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST should > he a run-time choice. > On Oct 10, 2012 8:30 AM, "Thomas Witkowski" < > thomas.witkowski at tu-dresden.de> wrote: > >> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use >> MatConvert, but it seems that this convert is not yet implemented. As I >> need this functionality to factorize the nested matrix with MUMPS, is there >> any better way than to extract the nested matrices, iterate over all rows >> of these matrices and use MatSetValues to add them again to the final >> MatAIJ? >> >> Thomas >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 10 08:50:49 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 08:50:49 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: <50757C05.20406@tu-dresden.de> References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> Message-ID: How did you get the MatNest? What you describe is the right approach, but proper preallocation is messy in the general case. There is a partial implementation in petsc-dev, but it needs to be finished/debugged. On Oct 10, 2012 8:45 AM, "Thomas Witkowski" wrote: > You are right. But what if a have a MATNEST and i must convert it to > MATAIJ. Any better way than what I wrote in my first mail? > > Thomas > > Am 10.10.2012 15:33, schrieb Jed Brown: > > The best way, by far, is to assemble the matrix as AIJ if that is what you > want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST should > he a run-time choice. > On Oct 10, 2012 8:30 AM, "Thomas Witkowski" < > thomas.witkowski at tu-dresden.de> wrote: > >> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use >> MatConvert, but it seems that this convert is not yet implemented. As I >> need this functionality to factorize the nested matrix with MUMPS, is there >> any better way than to extract the nested matrices, iterate over all rows >> of these matrices and use MatSetValues to add them again to the final >> MatAIJ? >> >> Thomas >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 10 08:54:03 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 10 Oct 2012 15:54:03 +0200 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> Message-ID: <50757DFB.5010308@tu-dresden.de> No, I didn't miss Jed's point. I know exactly, what you are talking about. My point is, that I can either write a preamble of 100 lines, which nobody is interested in, why I MUST HAVE a MatNest in this case and than ask how to convert to a MatAIJ (as I mentioned, this conversion is for debugging only!), or I can just save my and your time and ask directly. But still thank you very much for spending your time to help me in using PETSc!!! Thomas Am 10.10.2012 15:48, schrieb Matthew Knepley: > On Wed, Oct 10, 2012 at 9:45 AM, Thomas Witkowski > > wrote: > > You are right. But what if a have a MATNEST and i must convert it > to MATAIJ. Any better way than what I wrote in my first mail? > > > I think you miss Jed's point. You should NEVER have a MATNEST. You > should have a Mat. Its type > should be determined by MatSetFromOptions(). If you want MATAIJ, you > give -mat_type aij, and if you > want MATNEST, you give -mat_type nest. > > Thanks, > > Matt > > Thomas > > Am 10.10.2012 15:33, schrieb Jed Brown: >> >> The best way, by far, is to assemble the matrix as AIJ if that is >> what you want to use. See MatGetLocalSubMatrix and SNES ex28. >> Using MATNEST should he a run-time choice. >> >> On Oct 10, 2012 8:30 AM, "Thomas Witkowski" >> > > wrote: >> >> Is there a fast way to convert a MatNest to a MatAIJ? I tried >> to use MatConvert, but it seems that this convert is not yet >> implemented. As I need this functionality to factorize the >> nested matrix with MUMPS, is there any better way than to >> extract the nested matrices, iterate over all rows of these >> matrices and use MatSetValues to add them again to the final >> MatAIJ? >> >> Thomas >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Oct 10 11:38:42 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Oct 2012 11:38:42 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> Message-ID: <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> On Oct 10, 2012, at 8:50 AM, Jed Brown wrote: > How did you get the MatNest? > > What you describe is the right approach, but proper preallocation is messy in the general case. There is a partial implementation in petsc-dev, but it needs to be finished/debugged. Thomas, We should have a reasonable MatConvert_MatNest_MATAIJ. Perhaps you can look at what Jed has started in petsc-dev and improve it enough to suit your purpose. Barry > > On Oct 10, 2012 8:45 AM, "Thomas Witkowski" wrote: > You are right. But what if a have a MATNEST and i must convert it to MATAIJ. Any better way than what I wrote in my first mail? > > Thomas > > Am 10.10.2012 15:33, schrieb Jed Brown: >> The best way, by far, is to assemble the matrix as AIJ if that is what you want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST should he a run-time choice. >> >> On Oct 10, 2012 8:30 AM, "Thomas Witkowski" wrote: >> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use MatConvert, but it seems that this convert is not yet implemented. As I need this functionality to factorize the nested matrix with MUMPS, is there any better way than to extract the nested matrices, iterate over all rows of these matrices and use MatSetValues to add them again to the final MatAIJ? >> >> Thomas > From karpeev at mcs.anl.gov Wed Oct 10 11:57:36 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 10 Oct 2012 11:57:36 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> Message-ID: On Wed, Oct 10, 2012 at 11:38 AM, Barry Smith wrote: > > On Oct 10, 2012, at 8:50 AM, Jed Brown wrote: > > > How did you get the MatNest? > > > > What you describe is the right approach, but proper preallocation is > messy in the general case. There is a partial implementation in petsc-dev, > but it needs to be finished/debugged. > > Thomas, > > We should have a reasonable MatConvert_MatNest_MATAIJ. Perhaps you > can look at what Jed has started in petsc-dev and improve it enough to suit > your purpose. > There is this commit from last summer: http://petsc.cs.iit.edu/petsc/petsc-dev/rev/5289fd118d48 It's untested, but I pushed it back then hoping someone could beat it into shape. I also have a patch that eliminates ISAllGather() in favor of PetscSF operations, but that's even less tested (if such a thing is possible). I can clean it up and push, if somebody wants to play with it. Dmitry. > > Barry > > > > > > > > On Oct 10, 2012 8:45 AM, "Thomas Witkowski" < > thomas.witkowski at tu-dresden.de> wrote: > > You are right. But what if a have a MATNEST and i must convert it to > MATAIJ. Any better way than what I wrote in my first mail? > > > > Thomas > > > > Am 10.10.2012 15:33, schrieb Jed Brown: > >> The best way, by far, is to assemble the matrix as AIJ if that is what > you want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST > should he a run-time choice. > >> > >> On Oct 10, 2012 8:30 AM, "Thomas Witkowski" < > thomas.witkowski at tu-dresden.de> wrote: > >> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use > MatConvert, but it seems that this convert is not yet implemented. As I > need this functionality to factorize the nested matrix with MUMPS, is there > any better way than to extract the nested matrices, iterate over all rows > of these matrices and use MatSetValues to add them again to the final > MatAIJ? > >> > >> Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Wed Oct 10 12:07:14 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 10 Oct 2012 12:07:14 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> Message-ID: On Wed, Oct 10, 2012 at 8:33 AM, Jed Brown wrote: > The best way, by far, is to assemble the matrix as AIJ if that is what you > want to use. See MatGetLocalSubMatrix and SNES ex28. Using MATNEST should > he a run-time choice. > The caveat here is that currently there is no good way (that I know of) to preallocate in a mattype-oblivious way. And there is no preallocation of matnest matrices. There are bits a pieces of experimental code sitting around petsc-dev that could probably be made into a preallocation mechanism for matnest, but that's still quite a bit of work. Dmitry. > On Oct 10, 2012 8:30 AM, "Thomas Witkowski" < > thomas.witkowski at tu-dresden.de> wrote: > >> Is there a fast way to convert a MatNest to a MatAIJ? I tried to use >> MatConvert, but it seems that this convert is not yet implemented. As I >> need this functionality to factorize the nested matrix with MUMPS, is there >> any better way than to extract the nested matrices, iterate over all rows >> of these matrices and use MatSetValues to add them again to the final >> MatAIJ? >> >> Thomas >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 10 14:05:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 14:05:03 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> Message-ID: On Wed, Oct 10, 2012 at 11:38 AM, Barry Smith wrote: > We should have a reasonable MatConvert_MatNest_MATAIJ. Perhaps you > can look at what Jed has started in petsc-dev and improve it enough to suit > your purpose. Dmitry wrote that function, but it needs to be tested. I have some uncommitted test code, but I wanted to figure out why the test was failing/fix it before pushing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Wed Oct 10 14:12:01 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 10 Oct 2012 14:12:01 -0500 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> Message-ID: It would be good to have a meaningful test case to motivate the further development. I initially wrote it to enable GAMG for KKT systems, but that is still on the back burner, although it might be coming back to the front, since elastic contact is becoming more important for MOOSE. With a good test case it might be easier for me to improve the PetscSF-based impl as well, as it avoids the unscalable ISAllGather(). Dmitry. On Wed, Oct 10, 2012 at 2:05 PM, Jed Brown wrote: > On Wed, Oct 10, 2012 at 11:38 AM, Barry Smith wrote: > >> We should have a reasonable MatConvert_MatNest_MATAIJ. Perhaps you >> can look at what Jed has started in petsc-dev and improve it enough to suit >> your purpose. > > > Dmitry wrote that function, but it needs to be tested. I have some > uncommitted test code, but I wanted to figure out why the test was > failing/fix it before pushing. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 10 14:12:21 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 10 Oct 2012 21:12:21 +0200 Subject: [petsc-users] Convert MatNest to MatAIJ In-Reply-To: References: <50757873.7010704@tu-dresden.de> <50757C05.20406@tu-dresden.de> <6B46AAE5-1AAB-4D1B-9DF4-0652ABD7B73F@mcs.anl.gov> Message-ID: <5075C895.7080702@tu-dresden.de> Am 10.10.2012 18:57, schrieb Dmitry Karpeev: > There is this commit from last summer: > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/5289fd118d48 > It's untested, but I pushed it back then hoping someone could beat it > into shape. > I also have a patch that eliminates ISAllGather() in favor of PetscSF > operations, but that's even > less tested (if such a thing is possible). I can clean it up and > push, if somebody wants to play with it. I think, this will help me to make something working for my case, thanks! Thomas > > Dmitry. > > > Barry > > > > > > > > On Oct 10, 2012 8:45 AM, "Thomas Witkowski" > > wrote: > > You are right. But what if a have a MATNEST and i must convert > it to MATAIJ. Any better way than what I wrote in my first mail? > > > > Thomas > > > > Am 10.10.2012 15:33, schrieb Jed Brown: > >> The best way, by far, is to assemble the matrix as AIJ if that > is what you want to use. See MatGetLocalSubMatrix and SNES ex28. > Using MATNEST should he a run-time choice. > >> > >> On Oct 10, 2012 8:30 AM, "Thomas Witkowski" > > wrote: > >> Is there a fast way to convert a MatNest to a MatAIJ? I tried > to use MatConvert, but it seems that this convert is not yet > implemented. As I need this functionality to factorize the nested > matrix with MUMPS, is there any better way than to extract the > nested matrices, iterate over all rows of these matrices and use > MatSetValues to add them again to the final MatAIJ? > >> > >> Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From panourg at mech.upatras.gr Wed Oct 10 15:32:04 2012 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Wed, 10 Oct 2012 23:32:04 +0300 (EEST) Subject: [petsc-users] snes + matrix free + coloring Message-ID: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> Dear all I have a non-linear system which i want to solve it with matrix free method. However i must form the whole proconditioning matrix using finite differences and coloring method. I can set the function F() of non-linear system (i take the notation F() from documentation). Could you send me exactly the commands and the order of it which i must use or an apt example for that? I have looked for that in internet but i could not find any suitable example. Kostas Thanks From jedbrown at mcs.anl.gov Wed Oct 10 15:35:17 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 15:35:17 -0500 Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> Message-ID: On Wed, Oct 10, 2012 at 3:32 PM, wrote: > Dear all > > I have a non-linear system which i want to solve it with matrix free > method. However i must form the whole proconditioning matrix using finite > differences and coloring method. > How is the preconditioning matrix different from the actual operator? > > I can set the function F() of non-linear system (i take the notation F() > from documentation). > > Could you send me exactly the commands and the order of it which i must > use or an apt example for that? > > I have looked for that in internet but i could not find any suitable > example. > > Kostas > > Thanks > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From panourg at mech.upatras.gr Wed Oct 10 15:44:02 2012 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Wed, 10 Oct 2012 23:44:02 +0300 (EEST) Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> Message-ID: <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> Is exactly the same. Kostas > On Wed, Oct 10, 2012 at 3:32 PM, wrote: > >> Dear all >> >> I have a non-linear system which i want to solve it with matrix free >> method. However i must form the whole proconditioning matrix using >> finite >> differences and coloring method. >> > > How is the preconditioning matrix different from the actual operator? > > >> >> I can set the function F() of non-linear system (i take the notation F() >> from documentation). >> >> Could you send me exactly the commands and the order of it which i must >> use or an apt example for that? >> >> I have looked for that in internet but i could not find any suitable >> example. >> >> Kostas >> >> Thanks >> >> > From jedbrown at mcs.anl.gov Wed Oct 10 15:55:55 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 15:55:55 -0500 Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> Message-ID: Then you aren't using a matrix-free method at all. Just don't call SNESSetJacobian and it will be done for you. Are you using a DM? If not, you will also need to create a MatFDColoring. The easiest way to do that, if you don't have a structured mesh, is to assemble the nonzero pattern of your operator. On Wed, Oct 10, 2012 at 3:44 PM, wrote: > Is exactly the same. > > Kostas > > > > > > > > On Wed, Oct 10, 2012 at 3:32 PM, wrote: > > > >> Dear all > >> > >> I have a non-linear system which i want to solve it with matrix free > >> method. However i must form the whole proconditioning matrix using > >> finite > >> differences and coloring method. > >> > > > > How is the preconditioning matrix different from the actual operator? > > > > > >> > >> I can set the function F() of non-linear system (i take the notation F() > >> from documentation). > >> > >> Could you send me exactly the commands and the order of it which i must > >> use or an apt example for that? > >> > >> I have looked for that in internet but i could not find any suitable > >> example. > >> > >> Kostas > >> > >> Thanks > >> > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From panourg at mech.upatras.gr Wed Oct 10 16:33:08 2012 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Thu, 11 Oct 2012 00:33:08 +0300 (EEST) Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> Message-ID: <33177.94.66.14.234.1349904788.squirrel@mail.mech.upatras.gr> What do you mean when you say that i do not use a matrix free method at all? With matrix-free method only the operator is "free". Not the preconditioning matrix. For this i always set -snes_mf_operator option. Either way is any way to have preconditioner without forming the its prec matrix?? Panourgias Kostas > Then you aren't using a matrix-free method at all. Just don't call > SNESSetJacobian and it will be done for you. Are you using a DM? If not, > you will also need to create a MatFDColoring. The easiest way to do that, > if you don't have a structured mesh, is to assemble the nonzero pattern of > your operator. > > On Wed, Oct 10, 2012 at 3:44 PM, wrote: > >> Is exactly the same. >> >> Kostas >> >> >> >> >> >> >> > On Wed, Oct 10, 2012 at 3:32 PM, wrote: >> > >> >> Dear all >> >> >> >> I have a non-linear system which i want to solve it with matrix free >> >> method. However i must form the whole proconditioning matrix using >> >> finite >> >> differences and coloring method. >> >> >> > >> > How is the preconditioning matrix different from the actual operator? >> > >> > >> >> >> >> I can set the function F() of non-linear system (i take the notation >> F() >> >> from documentation). >> >> >> >> Could you send me exactly the commands and the order of it which i >> must >> >> use or an apt example for that? >> >> >> >> I have looked for that in internet but i could not find any suitable >> >> example. >> >> >> >> Kostas >> >> >> >> Thanks >> >> >> >> >> > >> >> > From jedbrown at mcs.anl.gov Wed Oct 10 16:47:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 16:47:19 -0500 Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: <33177.94.66.14.234.1349904788.squirrel@mail.mech.upatras.gr> References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> <33177.94.66.14.234.1349904788.squirrel@mail.mech.upatras.gr> Message-ID: On Wed, Oct 10, 2012 at 4:33 PM, wrote: > What do you mean when you say that i do not use a matrix free method at > all? > > With matrix-free method only the operator is "free". Not the > preconditioning matrix. > Yes, but if you are assembling the Jacobian using coloring, but then only using it as a preconditioner and using a separate matrix-free finite differencing step to apply the operator, then you are just goofing off. Of course you can do it, but it doesn't make sense. > > For this i always set -snes_mf_operator option. > > Either way is any way to have preconditioner without forming the its prec > matrix?? > > Sure, but you have to understand the problem, e.g., multilevel structure, plane waves, an embedded discretization. If you want to use PETSc's automatic preconditioners instead of rolling your own, then you should assemble the operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From panourg at mech.upatras.gr Wed Oct 10 17:13:59 2012 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Thu, 11 Oct 2012 01:13:59 +0300 (EEST) Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> <33177.94.66.14.234.1349904788.squirrel@mail.mech.upatras.gr> Message-ID: <52508.94.66.14.234.1349907239.squirrel@mail.mech.upatras.gr> It makes sense because i want to compute the preconditioner only per some time steps of my finite element code whereas in other times step (non-linear systems) i dont compute the prec and only the mf works using the same prec from previous step. Finally, could you tell me a source with all these functions so that i can solve my problem? Thanks > On Wed, Oct 10, 2012 at 4:33 PM, wrote: > >> What do you mean when you say that i do not use a matrix free method at >> all? >> >> With matrix-free method only the operator is "free". Not the >> preconditioning matrix. >> > > Yes, but if you are assembling the Jacobian using coloring, but then only > using it as a preconditioner and using a separate matrix-free finite > differencing step to apply the operator, then you are just goofing off. Of > course you can do it, but it doesn't make sense. > > >> >> For this i always set -snes_mf_operator option. >> >> Either way is any way to have preconditioner without forming the its >> prec >> matrix?? >> >> > Sure, but you have to understand the problem, e.g., multilevel structure, > plane waves, an embedded discretization. If you want to use PETSc's > automatic preconditioners instead of rolling your own, then you should > assemble the operator. > From jedbrown at mcs.anl.gov Wed Oct 10 17:17:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 10 Oct 2012 17:17:58 -0500 Subject: [petsc-users] snes + matrix free + coloring In-Reply-To: <52508.94.66.14.234.1349907239.squirrel@mail.mech.upatras.gr> References: <37756.94.66.14.234.1349901124.squirrel@mail.mech.upatras.gr> <56511.94.66.14.234.1349901842.squirrel@mail.mech.upatras.gr> <33177.94.66.14.234.1349904788.squirrel@mail.mech.upatras.gr> <52508.94.66.14.234.1349907239.squirrel@mail.mech.upatras.gr> Message-ID: You can set this as the Jacobian routine. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESDefaultComputeJacobianColor.html This example, cited in the link above, creates a coloring. http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex45.c.html On Wed, Oct 10, 2012 at 5:13 PM, wrote: > It makes sense because i want to compute the preconditioner only per some > time steps of my finite element code whereas in other times step > (non-linear systems) i dont compute the prec and only the mf works using > the same prec from previous step. > > Finally, could you tell me a source with all these functions so that i can > solve my problem? > > Thanks > > > > > > On Wed, Oct 10, 2012 at 4:33 PM, wrote: > > > >> What do you mean when you say that i do not use a matrix free method at > >> all? > >> > >> With matrix-free method only the operator is "free". Not the > >> preconditioning matrix. > >> > > > > Yes, but if you are assembling the Jacobian using coloring, but then only > > using it as a preconditioner and using a separate matrix-free finite > > differencing step to apply the operator, then you are just goofing off. > Of > > course you can do it, but it doesn't make sense. > > > > > >> > >> For this i always set -snes_mf_operator option. > >> > >> Either way is any way to have preconditioner without forming the its > >> prec > >> matrix?? > >> > >> > > Sure, but you have to understand the problem, e.g., multilevel structure, > > plane waves, an embedded discretization. If you want to use PETSc's > > automatic preconditioners instead of rolling your own, then you should > > assemble the operator. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From panourg at mech.upatras.gr Wed Oct 10 18:42:23 2012 From: panourg at mech.upatras.gr (panourg at mech.upatras.gr) Date: Thu, 11 Oct 2012 02:42:23 +0300 (EEST) Subject: [petsc-users] matgetcoloring Message-ID: <59455.94.66.14.234.1349912543.squirrel@mail.mech.upatras.gr> Dear all In function MatGetColoring() which is the faster method of coloring? Also is there any parameter so that i set such a poor coloring technique but fast? thanks From bsmith at mcs.anl.gov Wed Oct 10 19:54:47 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 10 Oct 2012 19:54:47 -0500 Subject: [petsc-users] matgetcoloring In-Reply-To: <59455.94.66.14.234.1349912543.squirrel@mail.mech.upatras.gr> References: <59455.94.66.14.234.1349912543.squirrel@mail.mech.upatras.gr> Message-ID: On Oct 10, 2012, at 6:42 PM, panourg at mech.upatras.gr wrote: > Dear all > > In function MatGetColoring() which is the faster method of coloring? Also > is there any parameter so that i set such a poor coloring technique but > fast? > > thanks The coloring time is not likely to dominate at all. It is only done once compared to many Jacobian evaluations so you want the best coloring (just use the default). Barry From gmnnus at gmail.com Sat Oct 13 14:17:04 2012 From: gmnnus at gmail.com (George Mathew) Date: Sat, 13 Oct 2012 15:17:04 -0400 Subject: [petsc-users] Help with mpiexec Message-ID: I have petsc 3.3 and mpich2 version 1.4.1p1 installed. When I run the example ex2.c code in the petsc user manual, it runs one one machine format. But, if I chose to run on multiple nodes, it gives error. The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c. I compiled it and when run using the following command, it works. mpiexec -f machinefile -n 1 ./ex2 But, if I run it using the command mpiexec -f machinefile -n 2 ./ex2 I get the following error. [0]PETSC ERROR: PetscOptionsInsertFile() line 485 in src/sys/objects/options.c [0]PETSC ERROR: PetscOptionsInsert() line 621 in src/sys/objects/options.c [0]PETSC ERROR: PetscInitialize() line 751 in src/sys/objects/pinit.c PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 Can anyone help? Thanx in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Oct 13 15:19:07 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 13 Oct 2012 16:19:07 -0400 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 3:17 PM, George Mathew wrote: > I have petsc 3.3 and mpich2 version 1.4.1p1 installed. > When I run the example ex2.c code in the petsc user manual, it runs one > one machine format. But, if I chose to run on multiple nodes, it gives > error. > The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c. > I compiled it and when run using the following command, it works. > mpiexec -f machinefile -n 1 ./ex2 > But, if I run it using the command > mpiexec -f machinefile -n 2 ./ex2 > I get the following error. > It is very likely that you are running the code with the wrong mpiexec (from another MPI installation). I cannot tell which MPI you used with configure.log. Matt > [0]PETSC ERROR: PetscOptionsInsertFile() line 485 in > src/sys/objects/options.c > [0]PETSC ERROR: PetscOptionsInsert() line 621 in src/sys/objects/options.c > [0]PETSC ERROR: PetscInitialize() line 751 in src/sys/objects/pinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > > Can anyone help? Thanx in advance. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 13 15:31:04 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 13 Oct 2012 15:31:04 -0500 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley wrote: > On Sat, Oct 13, 2012 at 3:17 PM, George Mathew wrote: > >> I have petsc 3.3 and mpich2 version 1.4.1p1 installed. >> When I run the example ex2.c code in the petsc user manual, it runs one >> one machine format. But, if I chose to run on multiple nodes, it gives >> error. >> The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c. >> I compiled it and when run using the following command, it works. >> mpiexec -f machinefile -n 1 ./ex2 >> But, if I run it using the command >> mpiexec -f machinefile -n 2 ./ex2 >> I get the following error. >> > > It is very likely that you are running the code with the wrong mpiexec > (from another MPI installation). > I cannot tell which MPI you used with configure.log. > Can you also try 1. Run multiple processes on your local machine 2. Run with -skip_petscrc -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmnnus at gmail.com Sat Oct 13 15:53:42 2012 From: gmnnus at gmail.com (George Mathew) Date: Sat, 13 Oct 2012 16:53:42 -0400 Subject: [petsc-users] Help with mpiexec Message-ID: On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley > wrote: >* On Sat, Oct 13, 2012 at 3:17 PM, George Mathew > wrote:*>**>>* I have petsc 3.3 and mpich2 version 1.4.1p1 installed.*>>* When I run the example ex2.c code in the petsc user manual, it runs one*>>* one machine format. But, if I chose to run on multiple nodes, it gives*>>* error.*>>* The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c.*>>* I compiled it and when run using the following command, it works.*>>* mpiexec -f machinefile -n 1 ./ex2*>>* But, if I run it using the command*>>* mpiexec -f machinefile -n 2 ./ex2*>>* I get the following error.*>>**>**>* It is very likely that you are running the code with the wrong mpiexec*>* (from another MPI installation).*>* I cannot tell which MPI you used with configure.log.*>** >Can you also try >1. Run multiple processes on your local machine >2. Run with -skip_petscrc I have only one MPI installation. It is under /usr/local/bin. The following command works: mpiexec -n 4 ./ex2 So, locally, multiple processes must be running fine. How do I run with -skip_petscrc? Thanx. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 13 15:55:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 13 Oct 2012 15:55:19 -0500 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 3:53 PM, George Mathew wrote: > On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley > > wrote: > > >* On Sat, Oct 13, 2012 at 3:17 PM, George Mathew > wrote:*>**>>* I have petsc 3.3 and mpich2 version 1.4.1p1 installed.*>>* When I run the example ex2.c code in the petsc user manual, it runs one*>>* one machine format. But, if I chose to run on multiple nodes, it gives*>>* error.*>>* The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c.*>>* I compiled it and when run using the following command, it works.*>>* mpiexec -f machinefile -n 1 ./ex2*>>* But, if I run it using the command*>>* mpiexec -f machinefile -n 2 ./ex2*>>* I get the following error.*>>**>**>* It is very likely that you are running the code with the wrong mpiexec*>* (from another MPI installation).*>* I cannot tell which MPI you used with configure.log.*>** > >Can you also try > > >1. Run multiple processes on your local machine > > >2. Run with -skip_petscrc > > I have only one MPI installation. It is under /usr/local/bin. The following command works: > > mpiexec -n 4 ./ex2 > > So, locally, multiple processes must be running fine. > > How do I run with -skip_petscrc? > > mpiexec -n 4 -f machinefile ./ex2 -skip_petscrc -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmnnus at gmail.com Sat Oct 13 16:00:27 2012 From: gmnnus at gmail.com (George Mathew) Date: Sat, 13 Oct 2012 17:00:27 -0400 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 4:55 PM, Jed Brown wrote: > On Sat, Oct 13, 2012 at 3:53 PM, George Mathew wrote: > >> On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley > >> wrote: >> >> >* On Sat, Oct 13, 2012 at 3:17 PM, George Mathew > wrote:*>**>>* I have petsc 3.3 and mpich2 version 1.4.1p1 installed.*>>* When I run the example ex2.c code in the petsc user manual, it runs one*>>* one machine format. But, if I chose to run on multiple nodes, it gives*>>* error.*>>* The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c.*>>* I compiled it and when run using the following command, it works.*>>* mpiexec -f machinefile -n 1 ./ex2*>>* But, if I run it using the command*>>* mpiexec -f machinefile -n 2 ./ex2*>>* I get the following error.*>>**>**>* It is very likely that you are running the code with the wrong mpiexec*>* (from another MPI installation).*>* I cannot tell which MPI you used with configure.log.*>** >> >Can you also try >> >> >1. Run multiple processes on your local machine >> >> >2. Run with -skip_petscrc >> >> I have only one MPI installation. It is under /usr/local/bin. The following command works: >> >> mpiexec -n 4 ./ex2 >> >> So, locally, multiple processes must be running fine. >> >> How do I run with -skip_petscrc? >> >> > mpiexec -n 4 -f machinefile ./ex2 -skip_petscrc > Here is the message when run with skip_petscrc: $ mpiexec -f machinefile -n 4 ./ex2 -skip_petscrc [0]PETSC ERROR: PetscWorldIsSingleHost() line 99 in src/sys/utils/pdisplay.c [0]PETSC ERROR: PetscSetDisplay() line 125 in src/sys/utils/pdisplay.c [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 319 in src/sys/objects/init.c [0]PETSC ERROR: PetscInitialize() line 761 in src/sys/objects/pinit.c PETSC ERROR: Logging has not been enabled. You might have forgotten to call PetscInitialize(). application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Oct 13 16:01:54 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 14 Oct 2012 08:01:54 +1100 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sun, Oct 14, 2012 at 8:00 AM, George Mathew wrote: > On Sat, Oct 13, 2012 at 4:55 PM, Jed Brown wrote: > >> On Sat, Oct 13, 2012 at 3:53 PM, George Mathew wrote: >> >>> On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley > >>> wrote: >>> >>> >* On Sat, Oct 13, 2012 at 3:17 PM, George Mathew > wrote:*>**>>* I have petsc 3.3 and mpich2 version 1.4.1p1 installed.*>>* When I run the example ex2.c code in the petsc user manual, it runs one*>>* one machine format. But, if I chose to run on multiple nodes, it gives*>>* error.*>>* The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c.*>>* I compiled it and when run using the following command, it works.*>>* mpiexec -f machinefile -n 1 ./ex2*>>* But, if I run it using the command*>>* mpiexec -f machinefile -n 2 ./ex2*>>* I get the following error.*>>**>**>* It is very likely that you are running the code with the wrong mpiexec*>* (from another MPI installation).*>* I cannot tell which MPI you used with configure.log.*>** >>> >Can you also try >>> >>> >1. Run multiple processes on your local machine >>> >>> >2. Run with -skip_petscrc >>> >>> I have only one MPI installation. It is under /usr/local/bin. The following command works: >>> >>> mpiexec -n 4 ./ex2 >>> >>> So, locally, multiple processes must be running fine. >>> >>> How do I run with -skip_petscrc? >>> >>> >> mpiexec -n 4 -f machinefile ./ex2 -skip_petscrc >> > > Here is the message when run with skip_petscrc: > > $ mpiexec -f machinefile -n 4 ./ex2 -skip_petscrc > [0]PETSC ERROR: PetscWorldIsSingleHost() line 99 in > src/sys/utils/pdisplay.c > [0]PETSC ERROR: PetscSetDisplay() line 125 in src/sys/utils/pdisplay.c > [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 319 in > src/sys/objects/init.c > [0]PETSC ERROR: PetscInitialize() line 761 in src/sys/objects/pinit.c > PETSC ERROR: Logging has not been enabled. > You might have forgotten to call PetscInitialize(). > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > Can you run any MPI job with that machinefile? Your MPI installations on the different machines could be incompatible. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmnnus at gmail.com Sat Oct 13 16:07:34 2012 From: gmnnus at gmail.com (George Mathew) Date: Sat, 13 Oct 2012 17:07:34 -0400 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 5:01 PM, Matthew Knepley wrote: > On Sun, Oct 14, 2012 at 8:00 AM, George Mathew wrote: > >> On Sat, Oct 13, 2012 at 4:55 PM, Jed Brown wrote: >> >>> On Sat, Oct 13, 2012 at 3:53 PM, George Mathew wrote: >>> >>>> On Sat, Oct 13, 2012 at 3:19 PM, Matthew Knepley > >>>> wrote: >>>> >>>> >* On Sat, Oct 13, 2012 at 3:17 PM, George Mathew > wrote:*>**>>* I have petsc 3.3 and mpich2 version 1.4.1p1 installed.*>>* When I run the example ex2.c code in the petsc user manual, it runs one*>>* one machine format. But, if I chose to run on multiple nodes, it gives*>>* error.*>>* The example is the file ${PETSC}/src/ksp/ksp/examples/tutorials/ex2.c.*>>* I compiled it and when run using the following command, it works.*>>* mpiexec -f machinefile -n 1 ./ex2*>>* But, if I run it using the command*>>* mpiexec -f machinefile -n 2 ./ex2*>>* I get the following error.*>>**>**>* It is very likely that you are running the code with the wrong mpiexec*>* (from another MPI installation).*>* I cannot tell which MPI you used with configure.log.*>** >>>> >Can you also try >>>> >>>> >1. Run multiple processes on your local machine >>>> >>>> >2. Run with -skip_petscrc >>>> >>>> I have only one MPI installation. It is under /usr/local/bin. The following command works: >>>> >>>> mpiexec -n 4 ./ex2 >>>> >>>> So, locally, multiple processes must be running fine. >>>> >>>> How do I run with -skip_petscrc? >>>> >>>> >>> mpiexec -n 4 -f machinefile ./ex2 -skip_petscrc >>> >> >> Here is the message when run with skip_petscrc: >> >> $ mpiexec -f machinefile -n 4 ./ex2 -skip_petscrc >> [0]PETSC ERROR: PetscWorldIsSingleHost() line 99 in >> src/sys/utils/pdisplay.c >> [0]PETSC ERROR: PetscSetDisplay() line 125 in src/sys/utils/pdisplay.c >> [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 319 in >> src/sys/objects/init.c >> [0]PETSC ERROR: PetscInitialize() line 761 in src/sys/objects/pinit.c >> PETSC ERROR: Logging has not been enabled. >> You might have forgotten to call PetscInitialize(). >> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> > > Can you run any MPI job with that machinefile? Your MPI installations on > the > different machines could be incompatible. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > The following command works fine: mpiexec -f machinefile -n 12 hostname It displays the hostname from the 4 different machines that I have in the machinefile. The MPI installation was done from the same tar file on the 4 hosts running identical versions of linux. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 13 16:09:15 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 13 Oct 2012 16:09:15 -0500 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 4:07 PM, George Mathew wrote: > The following command works fine: > mpiexec -f machinefile -n 12 hostname Hostname does not communicate at all. Can you run a real MPI program, such as cpi.c that is included in most source MPI distributions? The first call to MPI_Bcast() is raising an error, which strongly indicates that the MPI was misconfigured. > > It displays the hostname from the 4 different machines that I have in the > machinefile. The MPI installation was done from the same tar file on the 4 > hosts running identical versions of linux. > Environment variables could still be different, for example. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paeanball at gmail.com Sat Oct 13 17:04:55 2012 From: paeanball at gmail.com (Bao Kai) Date: Sun, 14 Oct 2012 01:04:55 +0300 Subject: [petsc-users] On changing the tolerance for SNES and KSP Message-ID: Hi, all, I am a little unclear about the exact definition of the relative and exact tolerance or how the PETSc determines the convergence, while my problem is as follows. Generally, all my equations are solved by SNES and I use the physical units. The PC and KSP are bjacobi and bicgstab respectively. For the pressure equation, at the beginning, the domain is not so big, and the pressure are within a certain range. The default parameters works. Then I increase the size of the domain by 10 times, then the range of the pressure will be about 10 times bigger accordingly due to gravity. Then with the default parameters, the KSP failed in getting converged with reaching the maximum iteration No. . Could you please help to tell me how I should change the tolerance to fit the change of the domain? Thank you very much. Best Regards, Kai -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmnnus at gmail.com Sat Oct 13 17:07:02 2012 From: gmnnus at gmail.com (George Mathew) Date: Sat, 13 Oct 2012 18:07:02 -0400 Subject: [petsc-users] Help with mpiexec In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 5:09 PM, Jed Brown wrote: > On Sat, Oct 13, 2012 at 4:07 PM, George Mathew wrote: > >> The following command works fine: >> mpiexec -f machinefile -n 12 hostname > > > Hostname does not communicate at all. Can you run a real MPI program, such > as cpi.c that is included in most source MPI distributions? The first call > to MPI_Bcast() is raising an error, which strongly indicates that the MPI > was misconfigured. > > >> >> It displays the hostname from the 4 different machines that I have in the >> machinefile. The MPI installation was done from the same tar file on the 4 >> hosts running identical versions of linux. >> > > Environment variables could still be different, for example. > That's it. MPI issue. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 13 17:16:15 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 13 Oct 2012 17:16:15 -0500 Subject: [petsc-users] On changing the tolerance for SNES and KSP In-Reply-To: References: Message-ID: 1. Especially with multi-physics simulations, it's important to nondimensionalize. You don't have to be formal about it, but computing with units that are well-scaled is useful to remove inessential ill-conditioning. 2. Most PETSc examples volume-weight residuals so that they can remain of the same order as you refine the grid. 3. Run with -snes_converged_reason -snes_monitor -ksp_converged_reason -ksp_monitor_true_residual. That will give you the important information about how the solver is converging. If it's not clear what is happening, you can send the output here. 4. More tips here: http://scicomp.stackexchange.com/a/514/119 On Sat, Oct 13, 2012 at 5:04 PM, Bao Kai wrote: > Hi, all, > > I am a little unclear about the exact definition of the relative and exact > tolerance or how the PETSc determines the convergence, while my problem is > as follows. > > Generally, all my equations are solved by SNES and I use the physical > units. The PC and KSP are bjacobi and bicgstab respectively. For the > pressure equation, at the beginning, the domain is not so big, and the > pressure are within a certain range. The default parameters works. > > Then I increase the size of the domain by 10 times, then the range of the > pressure will be about 10 times bigger accordingly due to gravity. Then > with the default parameters, the KSP failed in getting converged with > reaching the maximum iteration No. . > > Could you please help to tell me how I should change the tolerance to fit > the change of the domain? > > Thank you very much. > > Best Regards, > Kai > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paeanball at gmail.com Sun Oct 14 07:14:27 2012 From: paeanball at gmail.com (Bao Kai) Date: Sun, 14 Oct 2012 15:14:27 +0300 Subject: [petsc-users] How to display the actual Jacobian generated by SNES with fdcoloring? Message-ID: Hi, all, I want to check the exact Jacobian generated by SNES with fdcoloring with some small size problems. I mean to display the actual matrix and rhs. Could you please help to tell me how I can do that? Thank you. Best Regards, Kai -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Oct 14 11:37:38 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 14 Oct 2012 11:37:38 -0500 Subject: [petsc-users] How to display the actual Jacobian generated by SNES with fdcoloring? In-Reply-To: References: Message-ID: On Sun, Oct 14, 2012 at 7:14 AM, Bao Kai wrote: > Hi, all, > > I want to check the exact Jacobian generated by SNES with fdcoloring with > some small size problems. I mean to display the actual matrix and rhs. > You can use -snes_type test -snes_test_display or -snes_compare_explicit to "compare" the assembled matrix to one computed using FD. If you combine -snes_fd -snes_compare_explicit, the two matrices will be the same, but both will be printed. You can also call SNESDefaultComputeJacobian() and MatView() directly from your code. > > Could you please help to tell me how I can do that? > > Thank you. > > Best Regards, > Kai > -------------- next part -------------- An HTML attachment was scrubbed... URL: From subramanya.g at gmail.com Mon Oct 15 19:38:11 2012 From: subramanya.g at gmail.com (Subramanya G) Date: Mon, 15 Oct 2012 17:38:11 -0700 Subject: [petsc-users] Threaded Petsc Message-ID: Hi , I need iterative solvers for some of my code and I was planning to use petsc for it. Owing to the fact that some other parts of my code aren't parallelizable, I can't use petsc in parallel. However, I see that petsc can do threads now.. I have a small question about this . Does this allow me to assemble a matrix in the main thread. and then when solve is called will petsc take care of using multiple threads? Will this give me any speed ups? Thanks/ Subramanya G Sadasiva, Graduate Research Assistant, Hierarchical Design and Characterization Laboratory, School of Mechanical Engineering, Purdue University. "The art of structure is where to put the holes" Robert Le Ricolais, 1894-1977 From bsmith at mcs.anl.gov Mon Oct 15 19:44:09 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 15 Oct 2012 19:44:09 -0500 Subject: [petsc-users] Threaded Petsc In-Reply-To: References: Message-ID: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: > Hi , > I need iterative solvers for some of my code and I was planning to use > petsc for it. Owing to the fact that some other parts of my code > aren't parallelizable, I can't use petsc in parallel. However, I see > that petsc can do threads now.. I have a small question about this . > Does this allow me to assemble a matrix in the main thread. and then > when solve is called will petsc take care of using multiple threads? Yes > Will this give me any speed ups? Yes You need to use petsc-dev for this functionality and let us know if you have difficulties. Barry > Thanks/ > Subramanya G Sadasiva, > > Graduate Research Assistant, > Hierarchical Design and Characterization Laboratory, > School of Mechanical Engineering, > Purdue University. > > "The art of structure is where to put the holes" > Robert Le Ricolais, 1894-1977 From subramanya.g at gmail.com Mon Oct 15 23:35:54 2012 From: subramanya.g at gmail.com (Subramanya G) Date: Mon, 15 Oct 2012 21:35:54 -0700 Subject: [petsc-users] Threaded Petsc In-Reply-To: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> Message-ID: Hi, When I try to run the example suggested with -dm_vector_type pthread , I get a unknown vector type error. Unknown vector type: pthread! Here are my configure lines Configure options --download-mpich --download-f-blas-lapack --download-scientificpython --download-fiat --download-generator --download-chaco --download-triangle --with-shared-libraries --download-metis --download-parmetis --download-umfpack --download-hypre --sharedLibraryFlags=-fPIC -shared --with-pthreadclasses I am using the latest Petsc Dev version. Does anybody have any idea what might be wrong? It built without any issues. Do I need to do anything else when I configure. Subramanya G Sadasiva, Graduate Research Assistant, Hierarchical Design and Characterization Laboratory, School of Mechanical Engineering, Purdue University. "The art of structure is where to put the holes" Robert Le Ricolais, 1894-1977 On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: > > On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: > >> Hi , >> I need iterative solvers for some of my code and I was planning to use >> petsc for it. Owing to the fact that some other parts of my code >> aren't parallelizable, I can't use petsc in parallel. However, I see >> that petsc can do threads now.. I have a small question about this . >> Does this allow me to assemble a matrix in the main thread. and then >> when solve is called will petsc take care of using multiple threads? > > Yes > >> Will this give me any speed ups? > > Yes > > You need to use petsc-dev for this functionality and let us know if you have difficulties. > > Barry > >> Thanks/ >> Subramanya G Sadasiva, >> >> Graduate Research Assistant, >> Hierarchical Design and Characterization Laboratory, >> School of Mechanical Engineering, >> Purdue University. >> >> "The art of structure is where to put the holes" >> Robert Le Ricolais, 1894-1977 > From bsmith at mcs.anl.gov Mon Oct 15 23:49:11 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 15 Oct 2012 23:49:11 -0500 Subject: [petsc-users] Threaded Petsc In-Reply-To: References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> Message-ID: <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: > Hi, > When I try to run the example suggested with -dm_vector_type pthread > , I get a unknown vector type error. That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. Barry > > Unknown vector type: pthread! > > > > Here are my configure lines > Configure options --download-mpich --download-f-blas-lapack > --download-scientificpython --download-fiat --download-generator > --download-chaco --download-triangle --with-shared-libraries > --download-metis --download-parmetis --download-umfpack > --download-hypre --sharedLibraryFlags=-fPIC -shared > --with-pthreadclasses > > > > I am using the latest Petsc Dev version. > Does anybody have any idea what might be wrong? It built without any > issues. Do I need to do anything else when I configure. > > Subramanya G Sadasiva, > > Graduate Research Assistant, > Hierarchical Design and Characterization Laboratory, > School of Mechanical Engineering, > Purdue University. > > "The art of structure is where to put the holes" > Robert Le Ricolais, 1894-1977 > > > On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >> >> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >> >>> Hi , >>> I need iterative solvers for some of my code and I was planning to use >>> petsc for it. Owing to the fact that some other parts of my code >>> aren't parallelizable, I can't use petsc in parallel. However, I see >>> that petsc can do threads now.. I have a small question about this . >>> Does this allow me to assemble a matrix in the main thread. and then >>> when solve is called will petsc take care of using multiple threads? >> >> Yes >> >>> Will this give me any speed ups? >> >> Yes >> >> You need to use petsc-dev for this functionality and let us know if you have difficulties. >> >> Barry >> >>> Thanks/ >>> Subramanya G Sadasiva, >>> >>> Graduate Research Assistant, >>> Hierarchical Design and Characterization Laboratory, >>> School of Mechanical Engineering, >>> Purdue University. >>> >>> "The art of structure is where to put the holes" >>> Robert Le Ricolais, 1894-1977 >> From subramanya.g at gmail.com Mon Oct 15 23:57:47 2012 From: subramanya.g at gmail.com (Subramanya G) Date: Mon, 15 Oct 2012 21:57:47 -0700 Subject: [petsc-users] Threaded Petsc In-Reply-To: <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> Message-ID: Hi Barry, These are the instructions that I found. Thanks. http://www.mcs.anl.gov/petsc/petsc-dev/docs/installation.html#threads Subramanya G Sadasiva, Graduate Research Assistant, Hierarchical Design and Characterization Laboratory, School of Mechanical Engineering, Purdue University. "The art of structure is where to put the holes" Robert Le Ricolais, 1894-1977 On Mon, Oct 15, 2012 at 9:49 PM, Barry Smith wrote: > > On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: > >> Hi, >> When I try to run the example suggested with -dm_vector_type pthread >> , I get a unknown vector type error. > > That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. > > Barry > > >> >> Unknown vector type: pthread! >> >> >> >> Here are my configure lines >> Configure options --download-mpich --download-f-blas-lapack >> --download-scientificpython --download-fiat --download-generator >> --download-chaco --download-triangle --with-shared-libraries >> --download-metis --download-parmetis --download-umfpack >> --download-hypre --sharedLibraryFlags=-fPIC -shared >> --with-pthreadclasses >> >> >> >> I am using the latest Petsc Dev version. >> Does anybody have any idea what might be wrong? It built without any >> issues. Do I need to do anything else when I configure. >> >> Subramanya G Sadasiva, >> >> Graduate Research Assistant, >> Hierarchical Design and Characterization Laboratory, >> School of Mechanical Engineering, >> Purdue University. >> >> "The art of structure is where to put the holes" >> Robert Le Ricolais, 1894-1977 >> >> >> On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >>> >>> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >>> >>>> Hi , >>>> I need iterative solvers for some of my code and I was planning to use >>>> petsc for it. Owing to the fact that some other parts of my code >>>> aren't parallelizable, I can't use petsc in parallel. However, I see >>>> that petsc can do threads now.. I have a small question about this . >>>> Does this allow me to assemble a matrix in the main thread. and then >>>> when solve is called will petsc take care of using multiple threads? >>> >>> Yes >>> >>>> Will this give me any speed ups? >>> >>> Yes >>> >>> You need to use petsc-dev for this functionality and let us know if you have difficulties. >>> >>> Barry >>> >>>> Thanks/ >>>> Subramanya G Sadasiva, >>>> >>>> Graduate Research Assistant, >>>> Hierarchical Design and Characterization Laboratory, >>>> School of Mechanical Engineering, >>>> Purdue University. >>>> >>>> "The art of structure is where to put the holes" >>>> Robert Le Ricolais, 1894-1977 >>> > From bsmith at mcs.anl.gov Mon Oct 15 23:59:43 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Mon, 15 Oct 2012 23:59:43 -0500 Subject: [petsc-users] Threaded Petsc In-Reply-To: References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> Message-ID: Ok, thanks. Shri will update them with the latest info tomorrow. Sorry for the confusion. (they are easier :-)). Barry On Oct 15, 2012, at 11:57 PM, Subramanya G wrote: > Hi Barry, > These are the instructions that I found. > Thanks. > > http://www.mcs.anl.gov/petsc/petsc-dev/docs/installation.html#threads > > Subramanya G Sadasiva, > > Graduate Research Assistant, > Hierarchical Design and Characterization Laboratory, > School of Mechanical Engineering, > Purdue University. > > "The art of structure is where to put the holes" > Robert Le Ricolais, 1894-1977 > > > On Mon, Oct 15, 2012 at 9:49 PM, Barry Smith wrote: >> >> On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: >> >>> Hi, >>> When I try to run the example suggested with -dm_vector_type pthread >>> , I get a unknown vector type error. >> >> That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. >> >> Barry >> >> >>> >>> Unknown vector type: pthread! >>> >>> >>> >>> Here are my configure lines >>> Configure options --download-mpich --download-f-blas-lapack >>> --download-scientificpython --download-fiat --download-generator >>> --download-chaco --download-triangle --with-shared-libraries >>> --download-metis --download-parmetis --download-umfpack >>> --download-hypre --sharedLibraryFlags=-fPIC -shared >>> --with-pthreadclasses >>> >>> >>> >>> I am using the latest Petsc Dev version. >>> Does anybody have any idea what might be wrong? It built without any >>> issues. Do I need to do anything else when I configure. >>> >>> Subramanya G Sadasiva, >>> >>> Graduate Research Assistant, >>> Hierarchical Design and Characterization Laboratory, >>> School of Mechanical Engineering, >>> Purdue University. >>> >>> "The art of structure is where to put the holes" >>> Robert Le Ricolais, 1894-1977 >>> >>> >>> On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >>>> >>>> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >>>> >>>>> Hi , >>>>> I need iterative solvers for some of my code and I was planning to use >>>>> petsc for it. Owing to the fact that some other parts of my code >>>>> aren't parallelizable, I can't use petsc in parallel. However, I see >>>>> that petsc can do threads now.. I have a small question about this . >>>>> Does this allow me to assemble a matrix in the main thread. and then >>>>> when solve is called will petsc take care of using multiple threads? >>>> >>>> Yes >>>> >>>>> Will this give me any speed ups? >>>> >>>> Yes >>>> >>>> You need to use petsc-dev for this functionality and let us know if you have difficulties. >>>> >>>> Barry >>>> >>>>> Thanks/ >>>>> Subramanya G Sadasiva, >>>>> >>>>> Graduate Research Assistant, >>>>> Hierarchical Design and Characterization Laboratory, >>>>> School of Mechanical Engineering, >>>>> Purdue University. >>>>> >>>>> "The art of structure is where to put the holes" >>>>> Robert Le Ricolais, 1894-1977 >>>> >> From abhyshr at mcs.anl.gov Tue Oct 16 11:10:03 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 16 Oct 2012 11:10:03 -0500 Subject: [petsc-users] Threaded Petsc In-Reply-To: References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> Message-ID: <64EBE351-E19E-464E-A5F1-381B5A5775FF@mcs.anl.gov> Subramanya, Threads can be used in petsc-dev via the thread communicator object "threadcomm". PETSc currently has three thread communicators available: i) nothread - This is the default and does not use any threads. ii) pthread - This one uses native lockfree thread pool implementation using POSIX threads iii) OpenMP - OpenMP manages the threadpool. Configuration: Configure with --with-pthreadclasses=1 and/or --with-openmp=1 to use pthreads or OpenMP. **Note: If you are planning to use the OpenMP thread communicator only on MacOS then you'll need to additionally configure with --with-pthreadclasses=1. This is because OpenMP threadprivate variables are not available on MacOS and we use pthread_get/setspecific() calls for thread private variables instead (which are only available when configured with --with-pthreadclasses). Usage: cd src/sys/objects/threadcomm/examples/tutorials make ex1 ./ex3 -threadcomm_type -threadcomm_nthreads < # of threads> Use -h to get the list of available options with threadcomm. Examples: There are several examples in src/sys/threadcomm/examples/tutorials/ .Take a look at these to see how the threadcomm interface is used. We currently have PETSc's vector (seq and mpi) operations threaded but haven't made it open yet. These operations are guarded by the flag PETSC_THREADCOMM_ACTIVE. In order to use the threaded vector operations define PETSC_THREADCOMM_ACTIVE in $PETSC_DIR/$PETSC_ARCH/include/petscconf.h I'll update the docs on the website. Shri On Oct 15, 2012, at 11:59 PM, Barry Smith wrote: > > Ok, thanks. Shri will update them with the latest info tomorrow. Sorry for the confusion. (they are easier :-)). > > Barry > > > On Oct 15, 2012, at 11:57 PM, Subramanya G wrote: > >> Hi Barry, >> These are the instructions that I found. >> Thanks. >> >> http://www.mcs.anl.gov/petsc/petsc-dev/docs/installation.html#threads >> >> Subramanya G Sadasiva, >> >> Graduate Research Assistant, >> Hierarchical Design and Characterization Laboratory, >> School of Mechanical Engineering, >> Purdue University. >> >> "The art of structure is where to put the holes" >> Robert Le Ricolais, 1894-1977 >> >> >> On Mon, Oct 15, 2012 at 9:49 PM, Barry Smith wrote: >>> >>> On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: >>> >>>> Hi, >>>> When I try to run the example suggested with -dm_vector_type pthread >>>> , I get a unknown vector type error. >>> >>> That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. >>> >>> Barry >>> >>> >>>> >>>> Unknown vector type: pthread! >>>> >>>> >>>> >>>> Here are my configure lines >>>> Configure options --download-mpich --download-f-blas-lapack >>>> --download-scientificpython --download-fiat --download-generator >>>> --download-chaco --download-triangle --with-shared-libraries >>>> --download-metis --download-parmetis --download-umfpack >>>> --download-hypre --sharedLibraryFlags=-fPIC -shared >>>> --with-pthreadclasses >>>> >>>> >>>> >>>> I am using the latest Petsc Dev version. >>>> Does anybody have any idea what might be wrong? It built without any >>>> issues. Do I need to do anything else when I configure. >>>> >>>> Subramanya G Sadasiva, >>>> >>>> Graduate Research Assistant, >>>> Hierarchical Design and Characterization Laboratory, >>>> School of Mechanical Engineering, >>>> Purdue University. >>>> >>>> "The art of structure is where to put the holes" >>>> Robert Le Ricolais, 1894-1977 >>>> >>>> >>>> On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >>>>> >>>>> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >>>>> >>>>>> Hi , >>>>>> I need iterative solvers for some of my code and I was planning to use >>>>>> petsc for it. Owing to the fact that some other parts of my code >>>>>> aren't parallelizable, I can't use petsc in parallel. However, I see >>>>>> that petsc can do threads now.. I have a small question about this . >>>>>> Does this allow me to assemble a matrix in the main thread. and then >>>>>> when solve is called will petsc take care of using multiple threads? >>>>> >>>>> Yes >>>>> >>>>>> Will this give me any speed ups? >>>>> >>>>> Yes >>>>> >>>>> You need to use petsc-dev for this functionality and let us know if you have difficulties. >>>>> >>>>> Barry >>>>> >>>>>> Thanks/ >>>>>> Subramanya G Sadasiva, >>>>>> >>>>>> Graduate Research Assistant, >>>>>> Hierarchical Design and Characterization Laboratory, >>>>>> School of Mechanical Engineering, >>>>>> Purdue University. >>>>>> >>>>>> "The art of structure is where to put the holes" >>>>>> Robert Le Ricolais, 1894-1977 >>>>> >>> > From subramanya.g at gmail.com Tue Oct 16 11:30:13 2012 From: subramanya.g at gmail.com (Subramanya G) Date: Tue, 16 Oct 2012 09:30:13 -0700 Subject: [petsc-users] Threaded Petsc In-Reply-To: <64EBE351-E19E-464E-A5F1-381B5A5775FF@mcs.anl.gov> References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> <64EBE351-E19E-464E-A5F1-381B5A5775FF@mcs.anl.gov> Message-ID: Hi Thanks very much for the info. Does this mean that all the internal vector and matrix operations are multi-threaded? If I have a sequential code with PETSC_THREADCOMM_ACTIVE set to on will it be faster than a purely sequential code, especially for the KSP routines , which I guess depend on AXPY type operations. Thanks Subramanya G Sadasiva, Graduate Research Assistant, Hierarchical Design and Characterization Laboratory, School of Mechanical Engineering, Purdue University. "The art of structure is where to put the holes" Robert Le Ricolais, 1894-1977 On Tue, Oct 16, 2012 at 9:10 AM, Shri wrote: > Subramanya, > > Threads can be used in petsc-dev via the thread communicator object "threadcomm". PETSc currently has three thread communicators available: > i) nothread - This is the default and does not use any threads. > ii) pthread - This one uses native lockfree thread pool implementation using POSIX threads > iii) OpenMP - OpenMP manages the threadpool. > > Configuration: > Configure with --with-pthreadclasses=1 and/or --with-openmp=1 to use pthreads or OpenMP. > > **Note: If you are planning to use the OpenMP thread communicator only on MacOS then you'll need to additionally configure with --with-pthreadclasses=1. This is because OpenMP threadprivate variables are not available on MacOS and we use pthread_get/setspecific() calls for thread private variables instead (which are only available when configured with --with-pthreadclasses). > > Usage: > cd src/sys/objects/threadcomm/examples/tutorials > make ex1 > ./ex3 -threadcomm_type -threadcomm_nthreads < # of threads> > > Use -h to get the list of available options with threadcomm. > > Examples: > There are several examples in src/sys/threadcomm/examples/tutorials/ .Take a look at these to see how the threadcomm interface is used. > > We currently have PETSc's vector (seq and mpi) operations threaded but haven't made it open yet. These operations are guarded by the flag PETSC_THREADCOMM_ACTIVE. In order to use the threaded vector operations define PETSC_THREADCOMM_ACTIVE in $PETSC_DIR/$PETSC_ARCH/include/petscconf.h > > I'll update the docs on the website. > > Shri > > On Oct 15, 2012, at 11:59 PM, Barry Smith wrote: >> >> Ok, thanks. Shri will update them with the latest info tomorrow. Sorry for the confusion. (they are easier :-)). >> >> Barry >> >> >> On Oct 15, 2012, at 11:57 PM, Subramanya G wrote: >> >>> Hi Barry, >>> These are the instructions that I found. >>> Thanks. >>> >>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/installation.html#threads >>> >>> Subramanya G Sadasiva, >>> >>> Graduate Research Assistant, >>> Hierarchical Design and Characterization Laboratory, >>> School of Mechanical Engineering, >>> Purdue University. >>> >>> "The art of structure is where to put the holes" >>> Robert Le Ricolais, 1894-1977 >>> >>> >>> On Mon, Oct 15, 2012 at 9:49 PM, Barry Smith wrote: >>>> >>>> On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: >>>> >>>>> Hi, >>>>> When I try to run the example suggested with -dm_vector_type pthread >>>>> , I get a unknown vector type error. >>>> >>>> That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. >>>> >>>> Barry >>>> >>>> >>>>> >>>>> Unknown vector type: pthread! >>>>> >>>>> >>>>> >>>>> Here are my configure lines >>>>> Configure options --download-mpich --download-f-blas-lapack >>>>> --download-scientificpython --download-fiat --download-generator >>>>> --download-chaco --download-triangle --with-shared-libraries >>>>> --download-metis --download-parmetis --download-umfpack >>>>> --download-hypre --sharedLibraryFlags=-fPIC -shared >>>>> --with-pthreadclasses >>>>> >>>>> >>>>> >>>>> I am using the latest Petsc Dev version. >>>>> Does anybody have any idea what might be wrong? It built without any >>>>> issues. Do I need to do anything else when I configure. >>>>> >>>>> Subramanya G Sadasiva, >>>>> >>>>> Graduate Research Assistant, >>>>> Hierarchical Design and Characterization Laboratory, >>>>> School of Mechanical Engineering, >>>>> Purdue University. >>>>> >>>>> "The art of structure is where to put the holes" >>>>> Robert Le Ricolais, 1894-1977 >>>>> >>>>> >>>>> On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >>>>>> >>>>>> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >>>>>> >>>>>>> Hi , >>>>>>> I need iterative solvers for some of my code and I was planning to use >>>>>>> petsc for it. Owing to the fact that some other parts of my code >>>>>>> aren't parallelizable, I can't use petsc in parallel. However, I see >>>>>>> that petsc can do threads now.. I have a small question about this . >>>>>>> Does this allow me to assemble a matrix in the main thread. and then >>>>>>> when solve is called will petsc take care of using multiple threads? >>>>>> >>>>>> Yes >>>>>> >>>>>>> Will this give me any speed ups? >>>>>> >>>>>> Yes >>>>>> >>>>>> You need to use petsc-dev for this functionality and let us know if you have difficulties. >>>>>> >>>>>> Barry >>>>>> >>>>>>> Thanks/ >>>>>>> Subramanya G Sadasiva, >>>>>>> >>>>>>> Graduate Research Assistant, >>>>>>> Hierarchical Design and Characterization Laboratory, >>>>>>> School of Mechanical Engineering, >>>>>>> Purdue University. >>>>>>> >>>>>>> "The art of structure is where to put the holes" >>>>>>> Robert Le Ricolais, 1894-1977 >>>>>> >>>> >> > From abhyshr at mcs.anl.gov Tue Oct 16 11:36:55 2012 From: abhyshr at mcs.anl.gov (Shri) Date: Tue, 16 Oct 2012 11:36:55 -0500 Subject: [petsc-users] Threaded Petsc In-Reply-To: References: <636EDF14-6341-4140-A5C0-C508166FCABE@mcs.anl.gov> <6DCF155A-855D-4254-8D6F-08F29BCE9248@mcs.anl.gov> <64EBE351-E19E-464E-A5F1-381B5A5775FF@mcs.anl.gov> Message-ID: On Oct 16, 2012, at 11:30 AM, Subramanya G wrote: > Hi > Thanks very much for the info. Does this mean that all the internal > vector and matrix operations are multi-threaded? All the vector operations are threaded. All the matrix operations aren't threaded yet (only MatMult and a few others). We are working on threading the matrix operations once we test the vector operations. > If I have a > sequential code with PETSC_THREADCOMM_ACTIVE set to on will it be > faster than a purely sequential code, especially for the KSP routines > , which I guess depend on AXPY type operations. Yes, you should expect some speedup for the vector operations and MatMult. Shri > Thanks > > Subramanya G Sadasiva, > > Graduate Research Assistant, > Hierarchical Design and Characterization Laboratory, > School of Mechanical Engineering, > Purdue University. > > "The art of structure is where to put the holes" > Robert Le Ricolais, 1894-1977 > > > On Tue, Oct 16, 2012 at 9:10 AM, Shri wrote: >> Subramanya, >> >> Threads can be used in petsc-dev via the thread communicator object "threadcomm". PETSc currently has three thread communicators available: >> i) nothread - This is the default and does not use any threads. >> ii) pthread - This one uses native lockfree thread pool implementation using POSIX threads >> iii) OpenMP - OpenMP manages the threadpool. >> >> Configuration: >> Configure with --with-pthreadclasses=1 and/or --with-openmp=1 to use pthreads or OpenMP. >> >> **Note: If you are planning to use the OpenMP thread communicator only on MacOS then you'll need to additionally configure with --with-pthreadclasses=1. This is because OpenMP threadprivate variables are not available on MacOS and we use pthread_get/setspecific() calls for thread private variables instead (which are only available when configured with --with-pthreadclasses). >> >> Usage: >> cd src/sys/objects/threadcomm/examples/tutorials >> make ex1 >> ./ex3 -threadcomm_type -threadcomm_nthreads < # of threads> >> >> Use -h to get the list of available options with threadcomm. >> >> Examples: >> There are several examples in src/sys/threadcomm/examples/tutorials/ .Take a look at these to see how the threadcomm interface is used. >> >> We currently have PETSc's vector (seq and mpi) operations threaded but haven't made it open yet. These operations are guarded by the flag PETSC_THREADCOMM_ACTIVE. In order to use the threaded vector operations define PETSC_THREADCOMM_ACTIVE in $PETSC_DIR/$PETSC_ARCH/include/petscconf.h >> >> I'll update the docs on the website. >> >> Shri >> >> On Oct 15, 2012, at 11:59 PM, Barry Smith wrote: >>> >>> Ok, thanks. Shri will update them with the latest info tomorrow. Sorry for the confusion. (they are easier :-)). >>> >>> Barry >>> >>> >>> On Oct 15, 2012, at 11:57 PM, Subramanya G wrote: >>> >>>> Hi Barry, >>>> These are the instructions that I found. >>>> Thanks. >>>> >>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/installation.html#threads >>>> >>>> Subramanya G Sadasiva, >>>> >>>> Graduate Research Assistant, >>>> Hierarchical Design and Characterization Laboratory, >>>> School of Mechanical Engineering, >>>> Purdue University. >>>> >>>> "The art of structure is where to put the holes" >>>> Robert Le Ricolais, 1894-1977 >>>> >>>> >>>> On Mon, Oct 15, 2012 at 9:49 PM, Barry Smith wrote: >>>>> >>>>> On Oct 15, 2012, at 11:35 PM, Subramanya G wrote: >>>>> >>>>>> Hi, >>>>>> When I try to run the example suggested with -dm_vector_type pthread >>>>>> , I get a unknown vector type error. >>>>> >>>>> That documentation is out of date. Please let us know where you saw this "suggestion" so we can remove it. >>>>> >>>>> Barry >>>>> >>>>> >>>>>> >>>>>> Unknown vector type: pthread! >>>>>> >>>>>> >>>>>> >>>>>> Here are my configure lines >>>>>> Configure options --download-mpich --download-f-blas-lapack >>>>>> --download-scientificpython --download-fiat --download-generator >>>>>> --download-chaco --download-triangle --with-shared-libraries >>>>>> --download-metis --download-parmetis --download-umfpack >>>>>> --download-hypre --sharedLibraryFlags=-fPIC -shared >>>>>> --with-pthreadclasses >>>>>> >>>>>> >>>>>> >>>>>> I am using the latest Petsc Dev version. >>>>>> Does anybody have any idea what might be wrong? It built without any >>>>>> issues. Do I need to do anything else when I configure. >>>>>> >>>>>> Subramanya G Sadasiva, >>>>>> >>>>>> Graduate Research Assistant, >>>>>> Hierarchical Design and Characterization Laboratory, >>>>>> School of Mechanical Engineering, >>>>>> Purdue University. >>>>>> >>>>>> "The art of structure is where to put the holes" >>>>>> Robert Le Ricolais, 1894-1977 >>>>>> >>>>>> >>>>>> On Mon, Oct 15, 2012 at 5:44 PM, Barry Smith wrote: >>>>>>> >>>>>>> On Oct 15, 2012, at 7:38 PM, Subramanya G wrote: >>>>>>> >>>>>>>> Hi , >>>>>>>> I need iterative solvers for some of my code and I was planning to use >>>>>>>> petsc for it. Owing to the fact that some other parts of my code >>>>>>>> aren't parallelizable, I can't use petsc in parallel. However, I see >>>>>>>> that petsc can do threads now.. I have a small question about this . >>>>>>>> Does this allow me to assemble a matrix in the main thread. and then >>>>>>>> when solve is called will petsc take care of using multiple threads? >>>>>>> >>>>>>> Yes >>>>>>> >>>>>>>> Will this give me any speed ups? >>>>>>> >>>>>>> Yes >>>>>>> >>>>>>> You need to use petsc-dev for this functionality and let us know if you have difficulties. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>>> Thanks/ >>>>>>>> Subramanya G Sadasiva, >>>>>>>> >>>>>>>> Graduate Research Assistant, >>>>>>>> Hierarchical Design and Characterization Laboratory, >>>>>>>> School of Mechanical Engineering, >>>>>>>> Purdue University. >>>>>>>> >>>>>>>> "The art of structure is where to put the holes" >>>>>>>> Robert Le Ricolais, 1894-1977 >>>>>>> >>>>> >>> >> From tibo at berkeley.edu Tue Oct 16 13:54:26 2012 From: tibo at berkeley.edu (tibo at berkeley.edu) Date: Tue, 16 Oct 2012 11:54:26 -0700 Subject: [petsc-users] Array of KSP contexts Message-ID: <06b04a6b3714c03fd84c16651952c1b0.squirrel@calmail.berkeley.edu> Hi, I was wondering if it was possible to create arrays of KSP contexts, and if so, if someone could redirect me to an example or more info (I was not able to find this in the petsc manual). Some explanation: at some point in my code, I would like each processor to compute the LU factorization of a few sparse matrices each (using either mumps or superLU interface to petsc), store this using the KSP/PC tools, and be able to reuse it later in the code to solve some linear systems. The number of matrices that each processor has to factorize is dynamic so I can not create a different KSP context from the start, ideally I would like to have an array of KSP that I can allocate dynamically (I am a fortran user btw). thank you for your help, Tibo From bsmith at mcs.anl.gov Tue Oct 16 14:06:38 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 16 Oct 2012 14:06:38 -0500 Subject: [petsc-users] Array of KSP contexts In-Reply-To: <06b04a6b3714c03fd84c16651952c1b0.squirrel@calmail.berkeley.edu> References: <06b04a6b3714c03fd84c16651952c1b0.squirrel@calmail.berkeley.edu> Message-ID: KSP *ksps; ierr = PetscMalloc(n*sizeof(KSP),&ksps); for (i=0; i Hi, > > I was wondering if it was possible to create arrays of KSP contexts, and > if so, if someone could redirect me to an example or more info (I was not > able to find this in the petsc manual). > > Some explanation: at some point in my code, I would like each processor > to compute the LU factorization of a few sparse matrices each (using > either mumps or superLU interface to petsc), store this using the KSP/PC > tools, and be able to reuse it later in the code to solve some linear > systems. > > The number of matrices that each processor has to factorize is dynamic so > I can not create a different KSP context from the start, ideally I would > like to have an array of KSP that I can allocate dynamically (I am a > fortran user btw). > > thank you for your help, > > Tibo > From thomas.witkowski at tu-dresden.de Wed Oct 17 03:46:05 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 17 Oct 2012 10:46:05 +0200 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error Message-ID: <507E704D.8050206@tu-dresden.de> My code makes use of the function MatTransposeMatMult, and usually it work fine! For some larger input data, it now stops with a lot of MPI errors: fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed MPIR_Barrier(82)...: MPI_Waitall(261): MPI_Waitall(count=9, req_array=0xa787ba0, status_array=0xa789240) failed MPI_Waitall(113): The supplied request in array element 8 was invalid (kind=0) Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed MPIR_Barrier(82)...: mpid_irecv_done(98): read from socket failed - request state:recv(pde)done Here is the stack print from the debugger: 6, MatTransposeMatMult (matrix.c:8907) 6, MatTransposeMatMult_MPIAIJ_MPIAIJ (mpimatmatmult.c:809) 6, MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ (mpimatmatmult.c:1136) 6, PetscGatherMessageLengths2 (mpimesg.c:213) 6, PMPI_Waitall 6, MPIR_Err_return_comm 6, MPID_Abort I use PETSc 3.3-p3. Any idea whether this is or could be related to some bug in PETSc or whether I make wrong use of the function in some way? Thomas From paul.cruise.paul at gmail.com Wed Oct 17 09:18:42 2012 From: paul.cruise.paul at gmail.com (Paul Cruise) Date: Wed, 17 Oct 2012 16:18:42 +0200 Subject: [petsc-users] PETSc Execution Error Message-ID: Hello PETSc Users, I have a problem while execution of a PETSc code, please find attached the error file. There are no errors during compilation. Could someone please look into it & advise the source of error. Regards, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: --------------------- Error Message ------------------------------------ --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c From jroman at dsic.upv.es Wed Oct 17 09:30:44 2012 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 17 Oct 2012 16:30:44 +0200 Subject: [petsc-users] PETSc Execution Error In-Reply-To: References: Message-ID: <6CDFE96D-E005-420E-B3CC-1EB34764EE89@dsic.upv.es> El 17/10/2012, a las 16:18, Paul Cruise escribi?: > Hello PETSc Users, > > I have a problem while execution of a PETSc code, please find attached the error file. There are no errors during compilation. Could someone please look into it & advise the source of error. > > Regards, > Paul > This is the same error you reported three weeks ago. Did you try our suggestions? Place a MatMult operation before the solve (and comment out the rest of the code). If the problem persists then you have a problem when creating the matrix. Check the values that you are passing in MatSetValues. Jose From ling.zou at inl.gov Wed Oct 17 09:47:08 2012 From: ling.zou at inl.gov (Zou (Non-US), Ling) Date: Wed, 17 Oct 2012 08:47:08 -0600 Subject: [petsc-users] PETSc Execution Error In-Reply-To: References: Message-ID: Well, I am not PETSc expert, but the 'Floating point exception!' error message seems to me that you are trying to do a divide zero operation (or something else). I had this kind of error message before when I assemble my Jacobian matrix and I made a mistake trying to do this, like, return a/b; while b is not correctly initialized, it is zero. I am not quite sure what exactly happen in your code, but I would like to capture such bugs first. Best, Ling On Wed, Oct 17, 2012 at 8:18 AM, Paul Cruise wrote: > Hello PETSc Users, > > I have a problem while execution of a PETSc code, please find attached the > error file. There are no errors during compilation. Could someone please > look into it & advise the source of error. > > Regards, > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 17 09:52:01 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 17 Oct 2012 09:52:01 -0500 Subject: [petsc-users] PETSc Execution Error In-Reply-To: <6CDFE96D-E005-420E-B3CC-1EB34764EE89@dsic.upv.es> References: <6CDFE96D-E005-420E-B3CC-1EB34764EE89@dsic.upv.es> Message-ID: On Wed, Oct 17, 2012 at 9:30 AM, Jose E. Roman wrote: > This is the same error you reported three weeks ago. > Did you try our suggestions? Place a MatMult operation before the solve > (and comment out the rest of the code). If the problem persists then you > have a problem when creating the matrix. Check the values that you are > passing in MatSetValues. > You can instantly locate the place where the floating point exception occurred, if you are using glibc or a couple special architectures, by running in a debugger and passing the command line option -fp_trap to your application. It will break exactly at the place where the exception first occurred. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Oct 17 10:50:51 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 17 Oct 2012 10:50:51 -0500 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: <507E704D.8050206@tu-dresden.de> References: <507E704D.8050206@tu-dresden.de> Message-ID: Thomas: Does this occur only for large matrices? Can you dump your matrices into petsc binary files (e.g., A.dat, B.dat) and send to us for debugging? Lately, we added a new implementation of MatTransposeMatMult() in petsc-dev which is shown much faster than released MatTransposeMatMult(). You might give it a try by 1. install petsc-dev (see http://www.mcs.anl.gov/petsc/developers/index.html ) 2. run your code with option '-mattransposematmult_viamatmatmult 1' Let us know what you get. Hong My code makes use of the function MatTransposeMatMult, and usually it work > fine! For some larger input data, it now stops with a lot of MPI errors: > > fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed > MPIR_Barrier(82)...: > MPI_Waitall(261): MPI_Waitall(count=9, req_array=0xa787ba0, > status_array=0xa789240) failed > MPI_Waitall(113): The supplied request in array element 8 was invalid > (kind=0) > Fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed > MPIR_Barrier(82)...: > mpid_irecv_done(98): read from socket failed - request state:recv(pde)done > > > Here is the stack print from the debugger: > > 6, MatTransposeMatMult (matrix.c:8907) > 6, MatTransposeMatMult_MPIAIJ_**MPIAIJ > (mpimatmatmult.c:809) > 6, MatTransposeMatMultSymbolic_**MPIAIJ_MPIAIJ > (mpimatmatmult.c:1136) > 6, PetscGatherMessageLengths2 (mpimesg.c:213) > 6, PMPI_Waitall > 6, MPIR_Err_return_comm > 6, MPID_Abort > > > I use PETSc 3.3-p3. Any idea whether this is or could be related to some > bug in PETSc or whether I make wrong use of the function in some way? > > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 17 11:05:58 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 17 Oct 2012 11:05:58 -0500 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: References: <507E704D.8050206@tu-dresden.de> Message-ID: On Wed, Oct 17, 2012 at 10:50 AM, Hong Zhang wrote: > Lately, we added a new implementation of MatTransposeMatMult() in petsc-dev > which is shown much faster than released MatTransposeMatMult(). > You might give it a try by > 1. install petsc-dev (see > http://www.mcs.anl.gov/petsc/developers/index.html) > 2. run your code with option '-mattransposematmult_viamatmatmult 1' > Let us know what you get. > For the time-being, this will only work with non-broken MPI datatypes. https://svn.open-mpi.org/trac/ompi/ticket/2656 https://svn.open-mpi.org/trac/ompi/ticket/1905 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Oct 17 13:45:12 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Oct 2012 13:45:12 -0500 Subject: [petsc-users] [petsc-maint #136411] PETSc Execution Error In-Reply-To: References: Message-ID: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> Paul, With this type of problem you need to get a handle on exactly where the floating point exception appeared. This is done by running the debug version of the code in a debugger and asking the debugger to catch floating point exceptions (how to do this depends on the debugger). Once you see exactly where the floating point except ion comes from (for example a divide by zero, square root of a negative number etc) that can lead to seeing how to avoid it. Note to developers, we should have an answer for this in PETSc FAQ. Barry On Oct 17, 2012, at 9:18 AM, Paul Cruise wrote: > [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc12 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc13 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc22 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at beginning of function: Parameter number 2! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /gpfs/home/swayamjyoti_s/slepc-3.3/src/eps/examples/tutorials/Test/ex1f on a arch-linu named merlinc25 by swayamjyoti_s Tue Oct 16 11:47:26 2012 > [0]PETSC ERROR: Libraries linked from /gpfs/home/swayamjyoti_s/petsc-3.3/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Tue Aug 21 15:47:21 2012 > [0]PETSC ERROR: Configure options --with-fc=ifort --download-f-blas-lapack --download-mpich > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatMult() line 2081 in /gpfs/home/swayamjyoti_s/petsc-3.3/src/mat/interface/matrix.c > [0]PETSC ERROR: STApply_Shift() line 41 in src/st/impls/shift/shift.c > [0]PETSC ERROR: STApply() line 67 in src/st/interface/stsolve.c > [0]PETSC ERROR: EPSFullLanczos() line 179 in src/eps/impls/krylov/krylov.c > [0]PETSC ERROR: EPSSolve_KrylovSchur_Symm() line 58 in src/eps/impls/krylov/krylovschur/ks-symm.c > [0]PETSC ERROR: EPSSolve() line 130 in src/eps/interface/solve.c > > Hello PETSc Users, > > I have a problem while execution of a PETSc code, please find attached the > error file. There are no errors during compilation. Could someone please > look into it & advise the source of error. > > Regards, > Paul > Hello PETSc Users, > > I have a problem while execution of a PETSc code, please find attached the error file. There are no errors during compilation. Could someone please look into it & advise the source of error. > > Regards, > Paul From sgu at anl.gov Wed Oct 17 14:02:04 2012 From: sgu at anl.gov (Shiyuan Gu) Date: Wed, 17 Oct 2012 14:02:04 -0500 Subject: [petsc-users] SNES line search Message-ID: Hi, I am using petsc-3.3-p2. For my particular equation, for the first few iterations, the residual norm is effectively reduced and the solution X is very close to the exact solution (the residual norm is e-17). Then in the next iteration, SNES_KSPSolve(....) returns a newton step very close to zero which is correct, but the line search after it fails, and the function SNESLSCheckLocalMin_Private(...) is called. In the function SNESLSCheckLocalMin_Private(), the solution vector X is rewritten to X=J^T*F where J is the Jacobian and F is the residual. Since F is very close to zero, X is also zero which is wrong. Is there a way to skip the line search?(if the line search is not performed, there would be no problem for this particular equation). How should we handle this situation? Also, after the solution vector X is rewritten by SNESLSCheckLocalMin_Private(), the new residual norm is not printed on screen (which is huge for this particular problem). Since the residual norm printed on the screen is going to machine accuracy, users might think that the SNESSolve converges nicely and the returned solution is correct. This seems a bit misleading in my opinion. Thanks. Shiyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 17 14:11:06 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 17 Oct 2012 14:11:06 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: > Hi, > I am using petsc-3.3-p2. For my particular equation, for the first > few iterations, the residual norm is effectively reduced and the solution X > is very close to the exact solution (the residual norm is e-17). Then in > the next iteration, SNES_KSPSolve(....) returns a newton step very close to > zero which is correct, but the line search after it fails, and the function > SNESLSCheckLocalMin_Private(...) is called. In the function > SNESLSCheckLocalMin_Private(), > the solution vector X is rewritten to X=J^T*F where J is the Jacobian and > F is the residual. Since F is very close to zero, X is also zero which is > wrong. > Is there a way to skip the line search?(if the line search is not > performed, there would be no problem for this particular equation). How > should we handle this situation? > You can turn off the line search with -snes_linesearch_type basic, but I recommend setting an appropriate atol and stol instead, to prevent attempting to solve the system more accurately than machine precision when given a good initial guess. > Also, after the solution vector X is rewritten by > SNESLSCheckLocalMin_Private(), the new residual norm is not printed on > screen (which is huge for this particular problem). Since the residual > norm printed on the screen is going to machine accuracy, users might think > that the SNESSolve converges nicely and the returned solution is correct. > This seems a bit misleading in my opinion. > Are you checking -snes_converged_reason? > > Thanks. > > Shiyuan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Wed Oct 17 14:38:13 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Wed, 17 Oct 2012 14:38:13 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: <14016705-A6B0-4846-8FA7-A89444A67890@mcs.anl.gov> On Oct 17, 2012, at 2:02 PM, Shiyuan Gu wrote: > Hi, > I am using petsc-3.3-p2. For my particular equation, for the first few iterations, the residual norm is effectively reduced and the solution X is very close to the exact solution (the residual norm is e-17). Then in the next iteration, SNES_KSPSolve(....) returns a newton step very close to zero which is correct, but the line search after it fails, and the function SNESLSCheckLocalMin_Private(...) is called. In the function SNESLSCheckLocalMin_Private(), > the solution vector X is rewritten to X=J^T*F where J is the Jacobian and F is the residual. Since F is very close to zero, X is also zero which is wrong. > Is there a way to skip the line search?(if the line search is not performed, there would be no problem for this particular equation). How should we handle this situation? > > Also, after the solution vector X is rewritten by SNESLSCheckLocalMin_Private(), SNESLSCheckLocalMin_Private() is just a "sanity check" to help understand why the line search may have gone wrong. It does not change or "rewrite" the solution, in fact it doesn't affect the solution process at all. You can run the code with -ksp_monitor_true_residual -ksp_converged_reason -snes_monitor -snes_linesearch_monitor -info and send all the output to petsc-maint at mcs.anl.gov and we'll take a look at it. Normally if the Newton step is very very small the algorithm will "jump out" as converged. Barry > the new residual norm is not printed on screen (which is huge for this particular problem). Since the residual norm printed on the screen is going to machine accuracy, users might think that the SNESSolve converges nicely and the returned solution is correct. This seems a bit misleading in my opinion. > > Thanks. > > Shiyuan > From karpeev at mcs.anl.gov Wed Oct 17 14:40:58 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 17 Oct 2012 14:40:58 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: > On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: > >> Hi, >> I am using petsc-3.3-p2. For my particular equation, for the first >> few iterations, the residual norm is effectively reduced and the solution X >> is very close to the exact solution (the residual norm is e-17). Then in >> the next iteration, SNES_KSPSolve(....) returns a newton step very close to >> zero which is correct, but the line search after it fails, and the function >> SNESLSCheckLocalMin_Private(...) is called. In the function >> SNESLSCheckLocalMin_Private(), >> the solution vector X is rewritten to X=J^T*F where J is the Jacobian and >> F is the residual. Since F is very close to zero, X is also zero which is >> wrong. >> Is there a way to skip the line search?(if the line search is not >> performed, there would be no problem for this particular equation). How >> should we handle this situation? >> > > You can turn off the line search with -snes_linesearch_type basic, but I > recommend setting an appropriate atol and stol instead, to prevent > attempting to solve the system more accurately than machine precision when > given a good initial guess. > Maybe it wouldn't be a bad idea to (a) not use the solution vector as a temporary in detecting the local minimum, (b) print the norm of the last solution before bailing out of the Newton loop? Dmitry. > > >> Also, after the solution vector X is rewritten by >> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >> screen (which is huge for this particular problem). Since the residual >> norm printed on the screen is going to machine accuracy, users might think >> that the SNESSolve converges nicely and the returned solution is correct. >> This seems a bit misleading in my opinion. >> > > Are you checking -snes_converged_reason? > > >> >> Thanks. >> >> Shiyuan >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Wed Oct 17 14:56:22 2012 From: prbrune at gmail.com (Peter Brune) Date: Wed, 17 Oct 2012 14:56:22 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 2:40 PM, Dmitry Karpeev wrote: > > > On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: > >> On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: >> >>> Hi, >>> I am using petsc-3.3-p2. For my particular equation, for the first >>> few iterations, the residual norm is effectively reduced and the solution X >>> is very close to the exact solution (the residual norm is e-17). Then in >>> the next iteration, SNES_KSPSolve(....) returns a newton step very close to >>> zero which is correct, but the line search after it fails, and the function >>> SNESLSCheckLocalMin_Private(...) is called. In the function >>> SNESLSCheckLocalMin_Private(), >>> the solution vector X is rewritten to X=J^T*F where J is the Jacobian >>> and F is the residual. Since F is very close to zero, X is also zero which >>> is wrong. >>> Is there a way to skip the line search?(if the line search is not >>> performed, there would be no problem for this particular equation). How >>> should we handle this situation? >>> >> >> You can turn off the line search with -snes_linesearch_type basic, but I >> recommend setting an appropriate atol and stol instead, to prevent >> attempting to solve the system more accurately than machine precision when >> given a good initial guess. >> > Maybe it wouldn't be a bad idea to (a) not use the solution vector as a > temporary in detecting the local minimum, > Dumping garbage into X upon failure appears to be my careless mistake. I'll fix it in 3.3. > > > (b) print the norm of the last solution before bailing out of the Newton > loop? > This can go into the -info output. - Peter > Dmitry. > >> >> >>> Also, after the solution vector X is rewritten by >>> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >>> screen (which is huge for this particular problem). Since the residual >>> norm printed on the screen is going to machine accuracy, users might think >>> that the SNESSolve converges nicely and the returned solution is correct. >>> This seems a bit misleading in my opinion. >>> >> >> Are you checking -snes_converged_reason? >> >> >>> >>> Thanks. >>> >>> Shiyuan >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 17 14:57:05 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 17 Oct 2012 21:57:05 +0200 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: References: <507E704D.8050206@tu-dresden.de> Message-ID: <507F0D91.7020305@tu-dresden.de> Am 17.10.2012 17:50, schrieb Hong Zhang: > Thomas: > > Does this occur only for large matrices? > Can you dump your matrices into petsc binary files > (e.g., A.dat, B.dat) and send to us for debugging? > > Lately, we added a new implementation of MatTransposeMatMult() in > petsc-dev > which is shown much faster than released MatTransposeMatMult(). > You might give it a try by > 1. install petsc-dev (see > http://www.mcs.anl.gov/petsc/developers/index.html) > 2. run your code with option '-mattransposematmult_viamatmatmult 1' > Let us know what you get. > I checked the problem with petsc-dev. Here, the code just hangs somewhere inside MatTransposeMatMult. I checked, what MatTranspose does on the corresponding matrix and the behavior is the same. I extracted the matrix from my simulations, its of size 123,432 x 1,533,726 and very sparse (2 to 8 nnzs per row). I'm sorry, but this is the smallest matrix where I found the problem (I will send the matrix file to petsc-maint). I wrote some small piece of code, that just reads the matrix and runs MatTranspose. With 1 mpi task, it works fine. With small number of mpi tasks (so around 8), I get the following error message: [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. [1]PETSC ERROR: [1] PetscSFReduceEnd line 1259 src/sys/sf/sf.c [1]PETSC ERROR: [1] MatTranspose_MPIAIJ line 2045 src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: [1] MatTranspose line 4341 src/mat/interface/matrix.c With 32 mpi tasks, which I also use in my simulation, the code hangs in MatTranspose. If there is something more I can do to help you finding the problem, please let me know! Thomas > Hong > > My code makes use of the function MatTransposeMatMult, and usually > it work fine! For some larger input data, it now stops with a lot > of MPI errors: > > fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed > MPIR_Barrier(82)...: > MPI_Waitall(261): MPI_Waitall(count=9, req_array=0xa787ba0, > status_array=0xa789240) failed > MPI_Waitall(113): The supplied request in array element 8 was > invalid (kind=0) > Fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(476)..: MPI_Barrier(comm=0x84000001) failed > MPIR_Barrier(82)...: > mpid_irecv_done(98): read from socket failed - request > state:recv(pde)done > > > Here is the stack print from the debugger: > > 6, MatTransposeMatMult (matrix.c:8907) > 6, MatTransposeMatMult_MPIAIJ_MPIAIJ > (mpimatmatmult.c:809) > 6, MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ > (mpimatmatmult.c:1136) > 6, PetscGatherMessageLengths2 (mpimesg.c:213) > 6, PMPI_Waitall > 6, MPIR_Err_return_comm > 6, MPID_Abort > > > I use PETSc 3.3-p3. Any idea whether this is or could be related > to some bug in PETSc or whether I make wrong use of the function > in some way? > > Thomas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Wed Oct 17 15:01:59 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 17 Oct 2012 15:01:59 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 2:56 PM, Peter Brune wrote: > > > On Wed, Oct 17, 2012 at 2:40 PM, Dmitry Karpeev wrote: > >> >> >> On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: >> >>> On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: >>> >>>> Hi, >>>> I am using petsc-3.3-p2. For my particular equation, for the first >>>> few iterations, the residual norm is effectively reduced and the solution X >>>> is very close to the exact solution (the residual norm is e-17). Then in >>>> the next iteration, SNES_KSPSolve(....) returns a newton step very close to >>>> zero which is correct, but the line search after it fails, and the function >>>> SNESLSCheckLocalMin_Private(...) is called. In the function >>>> SNESLSCheckLocalMin_Private(), >>>> the solution vector X is rewritten to X=J^T*F where J is the Jacobian >>>> and F is the residual. Since F is very close to zero, X is also zero which >>>> is wrong. >>>> Is there a way to skip the line search?(if the line search is not >>>> performed, there would be no problem for this particular equation). How >>>> should we handle this situation? >>>> >>> >>> You can turn off the line search with -snes_linesearch_type basic, but I >>> recommend setting an appropriate atol and stol instead, to prevent >>> attempting to solve the system more accurately than machine precision when >>> given a good initial guess. >>> >> Maybe it wouldn't be a bad idea to (a) not use the solution vector as a >> temporary in detecting the local minimum, >> > > Dumping garbage into X upon failure appears to be my careless mistake. > I'll fix it in 3.3. > > >> >> >> > (b) print the norm of the last solution before bailing out of the Newton >> loop? >> > > This can go into the -info output. > Why not (also) let SNESMonitor() be called on that last solution? Dmitry. > > > - Peter > > >> Dmitry. >> >>> >>> >>>> Also, after the solution vector X is rewritten by >>>> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >>>> screen (which is huge for this particular problem). Since the residual >>>> norm printed on the screen is going to machine accuracy, users might think >>>> that the SNESSolve converges nicely and the returned solution is correct. >>>> This seems a bit misleading in my opinion. >>>> >>> >>> Are you checking -snes_converged_reason? >>> >>> >>>> >>>> Thanks. >>>> >>>> Shiyuan >>>> >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 17 15:21:48 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 17 Oct 2012 15:21:48 -0500 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: <507F0D91.7020305@tu-dresden.de> References: <507E704D.8050206@tu-dresden.de> <507F0D91.7020305@tu-dresden.de> Message-ID: As I said in the last mail, for the time-being you have to use a non-broken MPI. I listed the tickets. Open MPI has had multiple reduced test cases for several years now, but they haven't done anything to fix it yet. I will write an alternate implementation of PetscSF that does not use one-sided, but until then, you need a working MPI for the new MatTranspose(). On Wed, Oct 17, 2012 at 2:57 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 17 15:28:18 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 17 Oct 2012 22:28:18 +0200 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: References: <507E704D.8050206@tu-dresden.de> <507F0D91.7020305@tu-dresden.de> Message-ID: <507F14E2.1070206@tu-dresden.de> Jed, Sorry, I oversight this mail. My other choice is to use mpich. Do you have any idea whether this will work? Thomas Am 17.10.2012 22:21, schrieb Jed Brown: > As I said in the last mail, for the time-being you have to use a > non-broken MPI. I listed the tickets. Open MPI has had multiple > reduced test cases for several years now, but they haven't done > anything to fix it yet. > > I will write an alternate implementation of PetscSF that does not use > one-sided, but until then, you need a working MPI for the new > MatTranspose(). > > On Wed, Oct 17, 2012 at 2:57 PM, Thomas Witkowski > > wrote: > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 17 15:29:26 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 17 Oct 2012 15:29:26 -0500 Subject: [petsc-users] MatTransposeMatMult ends up with an MPI error In-Reply-To: <507F14E2.1070206@tu-dresden.de> References: <507E704D.8050206@tu-dresden.de> <507F0D91.7020305@tu-dresden.de> <507F14E2.1070206@tu-dresden.de> Message-ID: Yes, it works fine with MPICH2. On Wed, Oct 17, 2012 at 3:28 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Sorry, I oversight this mail. My other choice is to use mpich. Do you have > any idea whether this will work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From prbrune at gmail.com Wed Oct 17 15:34:23 2012 From: prbrune at gmail.com (Peter Brune) Date: Wed, 17 Oct 2012 15:34:23 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 3:01 PM, Dmitry Karpeev wrote: > > > On Wed, Oct 17, 2012 at 2:56 PM, Peter Brune wrote: > >> >> >> On Wed, Oct 17, 2012 at 2:40 PM, Dmitry Karpeev wrote: >> >>> >>> >>> On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: >>> >>>> On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: >>>> >>>>> Hi, >>>>> I am using petsc-3.3-p2. For my particular equation, for the >>>>> first few iterations, the residual norm is effectively reduced and the >>>>> solution X is very close to the exact solution (the residual norm is e-17). >>>>> Then in the next iteration, SNES_KSPSolve(....) returns a newton step very >>>>> close to zero which is correct, but the line search after it fails, and the >>>>> function SNESLSCheckLocalMin_Private(...) is called. In the function >>>>> SNESLSCheckLocalMin_Private(), >>>>> the solution vector X is rewritten to X=J^T*F where J is the Jacobian >>>>> and F is the residual. Since F is very close to zero, X is also zero which >>>>> is wrong. >>>>> Is there a way to skip the line search?(if the line search is not >>>>> performed, there would be no problem for this particular equation). How >>>>> should we handle this situation? >>>>> >>>> >>>> You can turn off the line search with -snes_linesearch_type basic, but >>>> I recommend setting an appropriate atol and stol instead, to prevent >>>> attempting to solve the system more accurately than machine precision when >>>> given a good initial guess. >>>> >>> Maybe it wouldn't be a bad idea to (a) not use the solution vector as a >>> temporary in detecting the local minimum, >>> >> >> Dumping garbage into X upon failure appears to be my careless mistake. >> I'll fix it in 3.3. >> >> >>> >>> >>> >> (b) print the norm of the last solution before bailing out of the Newton >>> loop? >>> >> >> This can go into the -info output. >> > Why not (also) let SNESMonitor() be called on that last solution? > Oh, I interpreted "norm of the solution" literally rather than "norm of the function at that solution", which makes more sense. However, as it quits after failing the line search and doesn't transfer the failed solution over. This is indeed a divergence, so printing a failed norm probably isn't the right solution. You can see what the line search is doing with -snes_linesearch_monitor. We changed the behavior in petsc-dev a little bit with respect to this type of case, as we were seeing bad interaction between line search divergence in the case of small step sizes and small-normed solutions with the logic as it was in 3.3 and before. This could be that type of problem, where the solution is well converged and the line search cannot make more progress. In any case, I'll fix the problem with writing garbage into the solution. In order to truly diagnose this, output with -snes_linesearch_monitor and -snes_converged reason would be nice. - Peter > > Dmitry. > >> >> >> - Peter >> >> >>> Dmitry. >>> >>>> >>>> >>>>> Also, after the solution vector X is rewritten by >>>>> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >>>>> screen (which is huge for this particular problem). Since the residual >>>>> norm printed on the screen is going to machine accuracy, users might think >>>>> that the SNESSolve converges nicely and the returned solution is correct. >>>>> This seems a bit misleading in my opinion. >>>>> >>>> >>>> Are you checking -snes_converged_reason? >>>> >>>> >>>>> >>>>> Thanks. >>>>> >>>>> Shiyuan >>>>> >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brune at mcs.anl.gov Wed Oct 17 16:45:42 2012 From: brune at mcs.anl.gov (Peter Brune) Date: Wed, 17 Oct 2012 16:45:42 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: Fix for the garbage in X part is in. http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/9d679552ef45 - Peter On Wed, Oct 17, 2012 at 3:34 PM, Peter Brune wrote: > > > On Wed, Oct 17, 2012 at 3:01 PM, Dmitry Karpeev wrote: > >> >> >> On Wed, Oct 17, 2012 at 2:56 PM, Peter Brune wrote: >> >>> >>> >>> On Wed, Oct 17, 2012 at 2:40 PM, Dmitry Karpeev wrote: >>> >>>> >>>> >>>> On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: >>>> >>>>> On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: >>>>> >>>>>> Hi, >>>>>> I am using petsc-3.3-p2. For my particular equation, for the >>>>>> first few iterations, the residual norm is effectively reduced and the >>>>>> solution X is very close to the exact solution (the residual norm is e-17). >>>>>> Then in the next iteration, SNES_KSPSolve(....) returns a newton step very >>>>>> close to zero which is correct, but the line search after it fails, and the >>>>>> function SNESLSCheckLocalMin_Private(...) is called. In the function >>>>>> SNESLSCheckLocalMin_Private(), >>>>>> the solution vector X is rewritten to X=J^T*F where J is the Jacobian >>>>>> and F is the residual. Since F is very close to zero, X is also zero which >>>>>> is wrong. >>>>>> Is there a way to skip the line search?(if the line search is not >>>>>> performed, there would be no problem for this particular equation). How >>>>>> should we handle this situation? >>>>>> >>>>> >>>>> You can turn off the line search with -snes_linesearch_type basic, but >>>>> I recommend setting an appropriate atol and stol instead, to prevent >>>>> attempting to solve the system more accurately than machine precision when >>>>> given a good initial guess. >>>>> >>>> Maybe it wouldn't be a bad idea to (a) not use the solution vector as a >>>> temporary in detecting the local minimum, >>>> >>> >>> Dumping garbage into X upon failure appears to be my careless mistake. >>> I'll fix it in 3.3. >>> >>> >>>> >>>> >>>> >>> (b) print the norm of the last solution before bailing out of the Newton >>>> loop? >>>> >>> >>> This can go into the -info output. >>> >> Why not (also) let SNESMonitor() be called on that last solution? >> > > Oh, I interpreted "norm of the solution" literally rather than "norm of > the function at that solution", which makes more sense. However, as it > quits after failing the line search and doesn't transfer the failed > solution over. This is indeed a divergence, so printing a failed norm > probably isn't the right solution. You can see what the line search is > doing with -snes_linesearch_monitor. > > We changed the behavior in petsc-dev a little bit with respect to this > type of case, as we were seeing bad interaction between line search > divergence in the case of small step sizes and small-normed solutions with > the logic as it was in 3.3 and before. This could be that type of problem, > where the solution is well converged and the line search cannot make more > progress. > > In any case, I'll fix the problem with writing garbage into the solution. > > In order to truly diagnose this, output with -snes_linesearch_monitor and > -snes_converged reason would be nice. > > - Peter > > >> >> > Dmitry. >> >>> >>> >>> - Peter >>> >>> >>>> Dmitry. >>>> >>>>> >>>>> >>>>>> Also, after the solution vector X is rewritten by >>>>>> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >>>>>> screen (which is huge for this particular problem). Since the residual >>>>>> norm printed on the screen is going to machine accuracy, users might think >>>>>> that the SNESSolve converges nicely and the returned solution is correct. >>>>>> This seems a bit misleading in my opinion. >>>>>> >>>>> >>>>> Are you checking -snes_converged_reason? >>>>> >>>>> >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Shiyuan >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karpeev at mcs.anl.gov Wed Oct 17 16:51:15 2012 From: karpeev at mcs.anl.gov (Dmitry Karpeev) Date: Wed, 17 Oct 2012 16:51:15 -0500 Subject: [petsc-users] SNES line search In-Reply-To: References: Message-ID: Thanks. Dmitry. On Wed, Oct 17, 2012 at 4:45 PM, Peter Brune wrote: > Fix for the garbage in X part is in. > > http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/9d679552ef45 > > - Peter > > > On Wed, Oct 17, 2012 at 3:34 PM, Peter Brune wrote: > >> >> >> On Wed, Oct 17, 2012 at 3:01 PM, Dmitry Karpeev wrote: >> >>> >>> >>> On Wed, Oct 17, 2012 at 2:56 PM, Peter Brune wrote: >>> >>>> >>>> >>>> On Wed, Oct 17, 2012 at 2:40 PM, Dmitry Karpeev wrote: >>>> >>>>> >>>>> >>>>> On Wed, Oct 17, 2012 at 2:11 PM, Jed Brown wrote: >>>>> >>>>>> On Wed, Oct 17, 2012 at 2:02 PM, Shiyuan Gu wrote: >>>>>> >>>>>>> Hi, >>>>>>> I am using petsc-3.3-p2. For my particular equation, for the >>>>>>> first few iterations, the residual norm is effectively reduced and the >>>>>>> solution X is very close to the exact solution (the residual norm is e-17). >>>>>>> Then in the next iteration, SNES_KSPSolve(....) returns a newton step very >>>>>>> close to zero which is correct, but the line search after it fails, and the >>>>>>> function SNESLSCheckLocalMin_Private(...) is called. In the function >>>>>>> SNESLSCheckLocalMin_Private(), >>>>>>> the solution vector X is rewritten to X=J^T*F where J is the >>>>>>> Jacobian and F is the residual. Since F is very close to zero, X is also >>>>>>> zero which is wrong. >>>>>>> Is there a way to skip the line search?(if the line search is not >>>>>>> performed, there would be no problem for this particular equation). How >>>>>>> should we handle this situation? >>>>>>> >>>>>> >>>>>> You can turn off the line search with -snes_linesearch_type basic, >>>>>> but I recommend setting an appropriate atol and stol instead, to prevent >>>>>> attempting to solve the system more accurately than machine precision when >>>>>> given a good initial guess. >>>>>> >>>>> Maybe it wouldn't be a bad idea to (a) not use the solution vector as >>>>> a temporary in detecting the local minimum, >>>>> >>>> >>>> Dumping garbage into X upon failure appears to be my careless mistake. >>>> I'll fix it in 3.3. >>>> >>>> >>>>> >>>>> >>>>> >>>> (b) print the norm of the last solution before bailing out of the >>>>> Newton loop? >>>>> >>>> >>>> This can go into the -info output. >>>> >>> Why not (also) let SNESMonitor() be called on that last solution? >>> >> >> Oh, I interpreted "norm of the solution" literally rather than "norm of >> the function at that solution", which makes more sense. However, as it >> quits after failing the line search and doesn't transfer the failed >> solution over. This is indeed a divergence, so printing a failed norm >> probably isn't the right solution. You can see what the line search is >> doing with -snes_linesearch_monitor. >> >> We changed the behavior in petsc-dev a little bit with respect to this >> type of case, as we were seeing bad interaction between line search >> divergence in the case of small step sizes and small-normed solutions with >> the logic as it was in 3.3 and before. This could be that type of problem, >> where the solution is well converged and the line search cannot make more >> progress. >> >> In any case, I'll fix the problem with writing garbage into the solution. >> >> In order to truly diagnose this, output with -snes_linesearch_monitor and >> -snes_converged reason would be nice. >> >> - Peter >> >> >>> >>> >> Dmitry. >>> >>>> >>>> >>>> - Peter >>>> >>>> >>>>> Dmitry. >>>>> >>>>>> >>>>>> >>>>>>> Also, after the solution vector X is rewritten by >>>>>>> SNESLSCheckLocalMin_Private(), the new residual norm is not printed on >>>>>>> screen (which is huge for this particular problem). Since the residual >>>>>>> norm printed on the screen is going to machine accuracy, users might think >>>>>>> that the SNESSolve converges nicely and the returned solution is correct. >>>>>>> This seems a bit misleading in my opinion. >>>>>>> >>>>>> >>>>>> Are you checking -snes_converged_reason? >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> Shiyuan >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From petsc-maint at mcs.anl.gov Wed Oct 17 18:57:50 2012 From: petsc-maint at mcs.anl.gov (Matthew Knepley) Date: Wed, 17 Oct 2012 16:57:50 -0700 Subject: [petsc-users] [petsc-maint #136521] petsc and openmp on mac In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 4:51 PM, gaddiel ouaknin wrote: > Hi, > > Petsc was installed on my computer which is a mac, > > using qt creator it gives me the following error: > > targeted os version does not support use of local thread variables... > > Now i try without success to insert the openmp flag into pestc without any > success > > So how to successfully install petsc with the openmp flag on mac and > working with QT ? > I do not believe PETSc connected with QT in any way. What are you trying to do? If you want to use compiler flags, the relevant configure variables come up with ./configure --help. Usually you override the optional flags ./configure ----COPTFLAGS="-openmp" Matt > Thank you very much, > > Gaddiel > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Thu Oct 18 08:05:52 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 18 Oct 2012 15:05:52 +0200 Subject: [petsc-users] Error during building - ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG Message-ID: <507FFEB0.4050801@gmail.com> Hi, I compiled and build PETSc dev and there's no error. Platform is vs2005 + ifort + cygwin. However, when I tried to build and run ex1f, I got the error: $ make ex1f /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -c -MT -Z7 -fpp -I/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include -I/cygdriv e/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include -I/cygdrive/d/wtay/Lib/MPICH2/include -o ex1f.o ex1f.F /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -MT -Z7 -fpp -o ex1f ex1f.o -L/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/lib - lpetsc -lflapack -lfblas /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2.lib /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2g.lib /cygdrive/d/wtay/Lib/MPICH2/lib/mpi.lib Gdi32. lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG D:\wtay\DOWNLO~1\Codes\PETSC-~1\src\ksp\ksp\examples\TUTORI~1\ex1f.exe : fatal error LNK1120: 1 unresolved externals makefile:26: recipe for target `ex1f' failed make: [ex1f] Error 96 (ignored) /usr/bin/rm -f ex1f.o I can't find the use of KSPMONITORLG so I don't understand where the error is coming from. Thanks for the help. -- Yours sincerely, TAY wee-beng From balay at mcs.anl.gov Thu Oct 18 08:25:16 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 18 Oct 2012 08:25:16 -0500 (CDT) Subject: [petsc-users] Error during building - ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG In-Reply-To: <507FFEB0.4050801@gmail.com> References: <507FFEB0.4050801@gmail.com> Message-ID: Hm - this issue was fixed yesterday. Perhaps you don't have latest petsc-dev? http://petsc.cs.iit.edu/petsc/petsc-dev/rev/dcba47e5e7e3 Satish On Thu, 18 Oct 2012, TAY wee-beng wrote: > Hi, > > I compiled and build PETSc dev and there's no error. Platform is vs2005 + > ifort + cygwin. > > However, when I tried to build and run ex1f, I got the error: > > $ make ex1f > /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -c > -MT -Z7 -fpp -I/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include > -I/cygdriv > e/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include > -I/cygdrive/d/wtay/Lib/MPICH2/include -o ex1f.o ex1f.F > /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -MT > -Z7 -fpp -o ex1f ex1f.o -L/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/lib > - > lpetsc -lflapack -lfblas /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2.lib > /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2g.lib > /cygdrive/d/wtay/Lib/MPICH2/lib/mpi.lib Gdi32. > lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib > ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG > D:\wtay\DOWNLO~1\Codes\PETSC-~1\src\ksp\ksp\examples\TUTORI~1\ex1f.exe : fatal > error LNK1120: 1 unresolved externals > makefile:26: recipe for target `ex1f' failed > make: [ex1f] Error 96 (ignored) > /usr/bin/rm -f ex1f.o > > I can't find the use of KSPMONITORLG so I don't understand where the error is > coming from. > > Thanks for the help. > > From gokhalen at gmail.com Thu Oct 18 13:07:21 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Thu, 18 Oct 2012 14:07:21 -0400 Subject: [petsc-users] Solving Ill Conditioned Problems Message-ID: Hi, I am solving a Piezoelectric problem using Petsc. The structure is [ K_uu K_uv ] [K_uv ^T -K_v,v ] More details about the formulation: http://tinyurl.com/9hlbp4u K_uu has elements O(1E9) because the stiffnesses are in GPa, K_uv has elements O(1) because piezoelectric coefficients are of that order K_v,v has elements O(1E-9) because the dielectric constants are of that order. I am using Petsc, with pc_type LU and MUMPS for the factorization, -ksp_type gmres. I am not sure if my solution is converging. A typical solve seems to be doing this: 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 Is there a right way to solve this set of equations? Is PCFieldSplit the recommended way? Thanks, -Nachiket From knepley at gmail.com Thu Oct 18 13:28:57 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 18 Oct 2012 11:28:57 -0700 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: References: Message-ID: On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale wrote: > Hi, > > I am solving a Piezoelectric problem using Petsc. The structure is > > [ K_uu K_uv ] > [K_uv ^T -K_v,v ] > > More details about the formulation: http://tinyurl.com/9hlbp4u > > K_uu has elements O(1E9) because the stiffnesses are in GPa, > K_uv has elements O(1) because piezoelectric coefficients are of that > order > K_v,v has elements O(1E-9) because the dielectric constants are of > that order. > > I am using Petsc, with pc_type LU and MUMPS for the factorization, > -ksp_type gmres. I am not sure if my solution is converging. A typical > solve seems to be doing this: > > 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm > 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 > 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm > 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 > 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm > 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 > 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm > 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 > > Is there a right way to solve this set of equations? Is PCFieldSplit > the recommended way? > First, you should really non-dimensionalize. You can see what this would give you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely everything will be O(1). Second, you might get something out of using FieldSplit, but its tough to tell without knowing more about the operators. Matt > Thanks, > > -Nachiket > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhalen at gmail.com Thu Oct 18 13:38:30 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Thu, 18 Oct 2012 14:38:30 -0400 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: References: Message-ID: On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: > On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale > wrote: >> >> Hi, >> >> I am solving a Piezoelectric problem using Petsc. The structure is >> >> [ K_uu K_uv ] >> [K_uv ^T -K_v,v ] >> >> More details about the formulation: http://tinyurl.com/9hlbp4u >> >> K_uu has elements O(1E9) because the stiffnesses are in GPa, >> K_uv has elements O(1) because piezoelectric coefficients are of that >> order >> K_v,v has elements O(1E-9) because the dielectric constants are of >> that order. >> >> I am using Petsc, with pc_type LU and MUMPS for the factorization, >> -ksp_type gmres. I am not sure if my solution is converging. A typical >> solve seems to be doing this: >> >> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >> >> Is there a right way to solve this set of equations? Is PCFieldSplit >> the recommended way? > > > First, you should really non-dimensionalize. You can see what this would > give > you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely > everything > will be O(1). > > Second, you might get something out of using FieldSplit, but its tough to > tell > without knowing more about the operators. > > Matt > >> >> Thanks, >> >> -Nachiket > Oh, right, thanks. I wasn't even thinking of it that way. I'll scale the variables and I'll give it a try. Cheers, -Nachiket > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener From zonexo at gmail.com Thu Oct 18 15:56:29 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Thu, 18 Oct 2012 22:56:29 +0200 Subject: [petsc-users] Error during building - ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG In-Reply-To: References: <507FFEB0.4050801@gmail.com> Message-ID: <50806CFC.9050906@gmail.com> Ok it's working now. Tks. Yours sincerely, TAY wee-beng On 18/10/2012 3:25 PM, Satish Balay wrote: > Hm - this issue was fixed yesterday. Perhaps you don't have latest petsc-dev? > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/dcba47e5e7e3 > > Satish > > On Thu, 18 Oct 2012, TAY wee-beng wrote: > >> Hi, >> >> I compiled and build PETSc dev and there's no error. Platform is vs2005 + >> ifort + cygwin. >> >> However, when I tried to build and run ex1f, I got the error: >> >> $ make ex1f >> /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -c >> -MT -Z7 -fpp -I/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include >> -I/cygdriv >> e/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/include >> -I/cygdrive/d/wtay/Lib/MPICH2/include -o ex1f.o ex1f.F >> /cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/bin/win32fe/win32fe ifort -MT >> -Z7 -fpp -o ex1f ex1f.o -L/cygdrive/d/wtay/Lib/petsc-3.3-dev_win32_vs2005/lib >> - >> lpetsc -lflapack -lfblas /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2.lib >> /cygdrive/d/wtay/Lib/MPICH2/lib/fmpich2g.lib >> /cygdrive/d/wtay/Lib/MPICH2/lib/mpi.lib Gdi32. >> lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib >> ex1f.obj : error LNK2001: unresolved external symbol KSPMONITORLG >> D:\wtay\DOWNLO~1\Codes\PETSC-~1\src\ksp\ksp\examples\TUTORI~1\ex1f.exe : fatal >> error LNK1120: 1 unresolved externals >> makefile:26: recipe for target `ex1f' failed >> make: [ex1f] Error 96 (ignored) >> /usr/bin/rm -f ex1f.o >> >> I can't find the use of KSPMONITORLG so I don't understand where the error is >> coming from. >> >> Thanks for the help. >> >> From popov at uni-mainz.de Fri Oct 19 11:02:57 2012 From: popov at uni-mainz.de (Anton Popov) Date: Fri, 19 Oct 2012 18:02:57 +0200 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: References: Message-ID: <508179B1.8010301@uni-mainz.de> Of, course you have a more sophisticated case, but if you just put the orders of magnitude of your matrix blocks into a toy two-by-two matrix, i.e. (in matlab notation) A = [1e9, 1; 1, 1e-9] then it's immediately visible that A is singular (det(A) is zero) Symmetric scaling, that Matt suggested, just makes it more obvious, after scaling the matrix A will be: A = [1, 1; 1, 1] - clearly the rows are linearly dependent. Scaling may help you, but of course, this is not an always working solution Anton On 10/18/12 8:38 PM, Nachiket Gokhale wrote: > On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: >> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >> wrote: >>> Hi, >>> >>> I am solving a Piezoelectric problem using Petsc. The structure is >>> >>> [ K_uu K_uv ] >>> [K_uv ^T -K_v,v ] >>> >>> More details about the formulation: http://tinyurl.com/9hlbp4u >>> >>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>> K_uv has elements O(1) because piezoelectric coefficients are of that >>> order >>> K_v,v has elements O(1E-9) because the dielectric constants are of >>> that order. >>> >>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>> solve seems to be doing this: >>> >>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>> >>> Is there a right way to solve this set of equations? Is PCFieldSplit >>> the recommended way? >> >> First, you should really non-dimensionalize. You can see what this would >> give >> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >> everything >> will be O(1). >> >> Second, you might get something out of using FieldSplit, but its tough to >> tell >> without knowing more about the operators. >> >> Matt >> >>> Thanks, >>> >>> -Nachiket > Oh, right, thanks. I wasn't even thinking of it that way. I'll scale > the variables and I'll give it a try. > > Cheers, > > -Nachiket > > > >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener From knepley at gmail.com Fri Oct 19 11:06:33 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 19 Oct 2012 09:06:33 -0700 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: <508179B1.8010301@uni-mainz.de> References: <508179B1.8010301@uni-mainz.de> Message-ID: On Fri, Oct 19, 2012 at 9:02 AM, Anton Popov wrote: > Of, course you have a more sophisticated case, but if you just put the > orders of magnitude of your matrix blocks > into a toy two-by-two matrix, i.e. (in matlab notation) > A = [1e9, 1; 1, 1e-9] > then it's immediately visible that A is singular (det(A) is zero) > Symmetric scaling, that Matt suggested, just makes it more obvious, after > scaling the matrix A will be: > A = [1, 1; 1, 1] - clearly the rows are linearly dependent. > Scaling may help you, but of course, this is not an always working solution > These are just the scales, not the values. Thus the matrix need not be singular. Matt > Anton > > On 10/18/12 8:38 PM, Nachiket Gokhale wrote: > >> On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley >> wrote: >> >>> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >>> wrote: >>> >>>> Hi, >>>> >>>> I am solving a Piezoelectric problem using Petsc. The structure is >>>> >>>> [ K_uu K_uv ] >>>> [K_uv ^T -K_v,v ] >>>> >>>> More details about the formulation: http://tinyurl.com/9hlbp4u >>>> >>>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>>> K_uv has elements O(1) because piezoelectric coefficients are of that >>>> order >>>> K_v,v has elements O(1E-9) because the dielectric constants are of >>>> that order. >>>> >>>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>>> solve seems to be doing this: >>>> >>>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>>> >>>> Is there a right way to solve this set of equations? Is PCFieldSplit >>>> the recommended way? >>>> >>> >>> First, you should really non-dimensionalize. You can see what this would >>> give >>> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >>> everything >>> will be O(1). >>> >>> Second, you might get something out of using FieldSplit, but its tough to >>> tell >>> without knowing more about the operators. >>> >>> Matt >>> >>> Thanks, >>>> >>>> -Nachiket >>>> >>> Oh, right, thanks. I wasn't even thinking of it that way. I'll scale >> the variables and I'll give it a try. >> >> Cheers, >> >> -Nachiket >> >> >> >> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments >>> is infinitely more interesting than any results to which their >>> experiments >>> lead. >>> -- Norbert Wiener >>> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Fri Oct 19 14:04:00 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Oct 2012 14:04:00 -0500 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: <508179B1.8010301@uni-mainz.de> References: <508179B1.8010301@uni-mainz.de> Message-ID: On Oct 19, 2012, at 11:02 AM, Anton Popov wrote: > Of, course you have a more sophisticated case, but if you just put the orders of magnitude of your matrix blocks > into a toy two-by-two matrix, i.e. (in matlab notation) > A = [1e9, 1; 1, 1e-9] > then it's immediately visible that A is singular (det(A) is zero) > Symmetric scaling, that Matt suggested, just makes it more obvious, after scaling the matrix A will be: > A = [1, 1; 1, 1] - clearly the rows are linearly dependent. > Scaling may help you, but of course, this is not an always working solution You need to do the scaling BEFORE you construct the matrix (or ideally before you do computing). Yes, forming the matrix and then scaling it is hopeless Barry > > Anton > > On 10/18/12 8:38 PM, Nachiket Gokhale wrote: >> On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: >>> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >>> wrote: >>>> Hi, >>>> >>>> I am solving a Piezoelectric problem using Petsc. The structure is >>>> >>>> [ K_uu K_uv ] >>>> [K_uv ^T -K_v,v ] >>>> >>>> More details about the formulation: http://tinyurl.com/9hlbp4u >>>> >>>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>>> K_uv has elements O(1) because piezoelectric coefficients are of that >>>> order >>>> K_v,v has elements O(1E-9) because the dielectric constants are of >>>> that order. >>>> >>>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>>> solve seems to be doing this: >>>> >>>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>>> >>>> Is there a right way to solve this set of equations? Is PCFieldSplit >>>> the recommended way? >>> >>> First, you should really non-dimensionalize. You can see what this would >>> give >>> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >>> everything >>> will be O(1). >>> >>> Second, you might get something out of using FieldSplit, but its tough to >>> tell >>> without knowing more about the operators. >>> >>> Matt >>> >>>> Thanks, >>>> >>>> -Nachiket >> Oh, right, thanks. I wasn't even thinking of it that way. I'll scale >> the variables and I'll give it a try. >> >> Cheers, >> >> -Nachiket >> >> >> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments >>> is infinitely more interesting than any results to which their experiments >>> lead. >>> -- Norbert Wiener > From gokhalen at gmail.com Fri Oct 19 14:09:07 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Fri, 19 Oct 2012 15:09:07 -0400 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: References: <508179B1.8010301@uni-mainz.de> Message-ID: Yes, I am doing the scaling in the element routines for the matrix assembly. I had not realized the importance of scaling. I was under the impression that if vastly different scales are in different blocks of the matrix, then the preconditioners or balancing operations will take care of the scaling. To be more precise, I was under the impression that things should be ok as long as one is not adding or subtracting numbers of vastly different scales to each other. -Nachiket On Fri, Oct 19, 2012 at 3:04 PM, Barry Smith wrote: > > On Oct 19, 2012, at 11:02 AM, Anton Popov wrote: > >> Of, course you have a more sophisticated case, but if you just put the orders of magnitude of your matrix blocks >> into a toy two-by-two matrix, i.e. (in matlab notation) >> A = [1e9, 1; 1, 1e-9] >> then it's immediately visible that A is singular (det(A) is zero) >> Symmetric scaling, that Matt suggested, just makes it more obvious, after scaling the matrix A will be: >> A = [1, 1; 1, 1] - clearly the rows are linearly dependent. >> Scaling may help you, but of course, this is not an always working solution > > You need to do the scaling BEFORE you construct the matrix (or ideally before you do computing). Yes, forming the matrix and then scaling it is hopeless > > Barry > >> >> Anton >> >> On 10/18/12 8:38 PM, Nachiket Gokhale wrote: >>> On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: >>>> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >>>> wrote: >>>>> Hi, >>>>> >>>>> I am solving a Piezoelectric problem using Petsc. The structure is >>>>> >>>>> [ K_uu K_uv ] >>>>> [K_uv ^T -K_v,v ] >>>>> >>>>> More details about the formulation: http://tinyurl.com/9hlbp4u >>>>> >>>>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>>>> K_uv has elements O(1) because piezoelectric coefficients are of that >>>>> order >>>>> K_v,v has elements O(1E-9) because the dielectric constants are of >>>>> that order. >>>>> >>>>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>>>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>>>> solve seems to be doing this: >>>>> >>>>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>>>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>>>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>>>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>>>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>>>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>>>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>>>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>>>> >>>>> Is there a right way to solve this set of equations? Is PCFieldSplit >>>>> the recommended way? >>>> >>>> First, you should really non-dimensionalize. You can see what this would >>>> give >>>> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >>>> everything >>>> will be O(1). >>>> >>>> Second, you might get something out of using FieldSplit, but its tough to >>>> tell >>>> without knowing more about the operators. >>>> >>>> Matt >>>> >>>>> Thanks, >>>>> >>>>> -Nachiket >>> Oh, right, thanks. I wasn't even thinking of it that way. I'll scale >>> the variables and I'll give it a try. >>> >>> Cheers, >>> >>> -Nachiket >>> >>> >>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments >>>> is infinitely more interesting than any results to which their experiments >>>> lead. >>>> -- Norbert Wiener >> > From bsmith at mcs.anl.gov Fri Oct 19 14:30:02 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Oct 2012 14:30:02 -0500 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: References: <508179B1.8010301@uni-mainz.de> Message-ID: <39C28495-1AA0-4883-B575-5E69CC58C5D4@mcs.anl.gov> I would do the scaling in the model before ever writing any code. The problem is that when the numbers are of very different sizes you have to be careful with ANY computations (though, as note, adding and subtracting is the most dangerous). Barry On Oct 19, 2012, at 2:09 PM, Nachiket Gokhale wrote: > Yes, I am doing the scaling in the element routines for the matrix assembly. > > I had not realized the importance of scaling. I was under the > impression that if vastly different scales are in different blocks of > the matrix, then the preconditioners or balancing operations will take > care of the scaling. To be more precise, I was under the impression > that things should be ok as long as one is not adding or subtracting > numbers of vastly different scales to each other. > > -Nachiket > > On Fri, Oct 19, 2012 at 3:04 PM, Barry Smith wrote: >> >> On Oct 19, 2012, at 11:02 AM, Anton Popov wrote: >> >>> Of, course you have a more sophisticated case, but if you just put the orders of magnitude of your matrix blocks >>> into a toy two-by-two matrix, i.e. (in matlab notation) >>> A = [1e9, 1; 1, 1e-9] >>> then it's immediately visible that A is singular (det(A) is zero) >>> Symmetric scaling, that Matt suggested, just makes it more obvious, after scaling the matrix A will be: >>> A = [1, 1; 1, 1] - clearly the rows are linearly dependent. >>> Scaling may help you, but of course, this is not an always working solution >> >> You need to do the scaling BEFORE you construct the matrix (or ideally before you do computing). Yes, forming the matrix and then scaling it is hopeless >> >> Barry >> >>> >>> Anton >>> >>> On 10/18/12 8:38 PM, Nachiket Gokhale wrote: >>>> On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: >>>>> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> I am solving a Piezoelectric problem using Petsc. The structure is >>>>>> >>>>>> [ K_uu K_uv ] >>>>>> [K_uv ^T -K_v,v ] >>>>>> >>>>>> More details about the formulation: http://tinyurl.com/9hlbp4u >>>>>> >>>>>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>>>>> K_uv has elements O(1) because piezoelectric coefficients are of that >>>>>> order >>>>>> K_v,v has elements O(1E-9) because the dielectric constants are of >>>>>> that order. >>>>>> >>>>>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>>>>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>>>>> solve seems to be doing this: >>>>>> >>>>>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>>>>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>>>>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>>>>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>>>>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>>>>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>>>>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>>>>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>>>>> >>>>>> Is there a right way to solve this set of equations? Is PCFieldSplit >>>>>> the recommended way? >>>>> >>>>> First, you should really non-dimensionalize. You can see what this would >>>>> give >>>>> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >>>>> everything >>>>> will be O(1). >>>>> >>>>> Second, you might get something out of using FieldSplit, but its tough to >>>>> tell >>>>> without knowing more about the operators. >>>>> >>>>> Matt >>>>> >>>>>> Thanks, >>>>>> >>>>>> -Nachiket >>>> Oh, right, thanks. I wasn't even thinking of it that way. I'll scale >>>> the variables and I'll give it a try. >>>> >>>> Cheers, >>>> >>>> -Nachiket >>>> >>>> >>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their experiments >>>>> is infinitely more interesting than any results to which their experiments >>>>> lead. >>>>> -- Norbert Wiener >>> >> From mirzadeh at gmail.com Fri Oct 19 16:16:07 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 19 Oct 2012 14:16:07 -0700 Subject: [petsc-users] matrix reordering Message-ID: Hi guys, Quick question. *After* the matrix is setup, is it possible to ask PETSc to renumber the nodes internally in the KSPSolve phase to minimize communications for MatMul inside the solver? Can I use the MatPartitioning object for this or is that only intended to partition the mesh as a pre-processing step? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From renzhengyong at gmail.com Fri Oct 19 16:38:06 2012 From: renzhengyong at gmail.com (Renzhengyong) Date: Fri, 19 Oct 2012 23:38:06 +0200 Subject: [petsc-users] Solving Ill Conditioned Problems In-Reply-To: <39C28495-1AA0-4883-B575-5E69CC58C5D4@mcs.anl.gov> References: <508179B1.8010301@uni-mainz.de> <39C28495-1AA0-4883-B575-5E69CC58C5D4@mcs.anl.gov> Message-ID: My experience strongly support Barry's suggestion. Zhengyong Zhengyong Ren, Institute of Geophysics, Department of Geoscience, ETH Zurich, CH8092, Zurich, Switzerland. On 19.10.2012, at 21:30, Barry Smith wrote: > > I would do the scaling in the model before ever writing any code. The problem is that when the numbers are of very different sizes you have to be careful with ANY computations (though, as note, adding and subtracting is the most dangerous). > > Barry > > > On Oct 19, 2012, at 2:09 PM, Nachiket Gokhale wrote: > >> Yes, I am doing the scaling in the element routines for the matrix assembly. >> >> I had not realized the importance of scaling. I was under the >> impression that if vastly different scales are in different blocks of >> the matrix, then the preconditioners or balancing operations will take >> care of the scaling. To be more precise, I was under the impression >> that things should be ok as long as one is not adding or subtracting >> numbers of vastly different scales to each other. >> >> -Nachiket >> >> On Fri, Oct 19, 2012 at 3:04 PM, Barry Smith wrote: >>> >>> On Oct 19, 2012, at 11:02 AM, Anton Popov wrote: >>> >>>> Of, course you have a more sophisticated case, but if you just put the orders of magnitude of your matrix blocks >>>> into a toy two-by-two matrix, i.e. (in matlab notation) >>>> A = [1e9, 1; 1, 1e-9] >>>> then it's immediately visible that A is singular (det(A) is zero) >>>> Symmetric scaling, that Matt suggested, just makes it more obvious, after scaling the matrix A will be: >>>> A = [1, 1; 1, 1] - clearly the rows are linearly dependent. >>>> Scaling may help you, but of course, this is not an always working solution >>> >>> You need to do the scaling BEFORE you construct the matrix (or ideally before you do computing). Yes, forming the matrix and then scaling it is hopeless >>> >>> Barry >>> >>>> >>>> Anton >>>> >>>> On 10/18/12 8:38 PM, Nachiket Gokhale wrote: >>>>> On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley wrote: >>>>>> On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I am solving a Piezoelectric problem using Petsc. The structure is >>>>>>> >>>>>>> [ K_uu K_uv ] >>>>>>> [K_uv ^T -K_v,v ] >>>>>>> >>>>>>> More details about the formulation: http://tinyurl.com/9hlbp4u >>>>>>> >>>>>>> K_uu has elements O(1E9) because the stiffnesses are in GPa, >>>>>>> K_uv has elements O(1) because piezoelectric coefficients are of that >>>>>>> order >>>>>>> K_v,v has elements O(1E-9) because the dielectric constants are of >>>>>>> that order. >>>>>>> >>>>>>> I am using Petsc, with pc_type LU and MUMPS for the factorization, >>>>>>> -ksp_type gmres. I am not sure if my solution is converging. A typical >>>>>>> solve seems to be doing this: >>>>>>> >>>>>>> 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm >>>>>>> 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14 >>>>>>> 29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm >>>>>>> 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14 >>>>>>> 30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm >>>>>>> 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14 >>>>>>> 31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm >>>>>>> 2.447631111938e-04 ||r(i)||/||b|| 6.606955965570e-15 >>>>>>> >>>>>>> Is there a right way to solve this set of equations? Is PCFieldSplit >>>>>>> the recommended way? >>>>>> >>>>>> First, you should really non-dimensionalize. You can see what this would >>>>>> give >>>>>> you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely >>>>>> everything >>>>>> will be O(1). >>>>>> >>>>>> Second, you might get something out of using FieldSplit, but its tough to >>>>>> tell >>>>>> without knowing more about the operators. >>>>>> >>>>>> Matt >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -Nachiket >>>>> Oh, right, thanks. I wasn't even thinking of it that way. I'll scale >>>>> the variables and I'll give it a try. >>>>> >>>>> Cheers, >>>>> >>>>> -Nachiket >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their experiments >>>>>> is infinitely more interesting than any results to which their experiments >>>>>> lead. >>>>>> -- Norbert Wiener >>>> >>> > From bsmith at mcs.anl.gov Fri Oct 19 17:00:47 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Fri, 19 Oct 2012 17:00:47 -0500 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: Mohammad, We've thought about adding this kind of support this over the years but always came to the conclusion that "the right way to do it" is during the mesh setup step. Barry On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh wrote: > Hi guys, > > Quick question. *After* the matrix is setup, is it possible to ask PETSc to renumber the nodes internally in the KSPSolve phase to minimize communications for MatMul inside the solver? Can I use the MatPartitioning object for this or is that only intended to partition the mesh as a pre-processing step? > > Thanks From mirzadeh at gmail.com Fri Oct 19 17:48:28 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 19 Oct 2012 15:48:28 -0700 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: Thanks Barry. I think I agree with you that right way to do it is through mesh setup. As a matter of fact I have done that for some of my problems. Yet, I can think of situations where that kind of support might be beneficial, especially if someone (a.k.a me in this case!) is looking for a 'dirty and fast' fix. Anyway, I still believe that in long term, one should do the partitioning in a pre-processing step anyways and so there may not be much incentive for adding such a support. Thanks again, Mohammad On Fri, Oct 19, 2012 at 3:00 PM, Barry Smith wrote: > > Mohammad, > > We've thought about adding this kind of support this over the years > but always came to the conclusion that "the right way to do it" is during > the mesh setup step. > > Barry > > > On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh wrote: > > > Hi guys, > > > > Quick question. *After* the matrix is setup, is it possible to ask PETSc > to renumber the nodes internally in the KSPSolve phase to minimize > communications for MatMul inside the solver? Can I use the MatPartitioning > object for this or is that only intended to partition the mesh as a > pre-processing step? > > > > Thanks > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Fri Oct 19 17:57:09 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 19 Oct 2012 17:57:09 -0500 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: _You_ can compute a MatOrdering, MatPermute and VecPermute, solve, and permute back. Profile and if the solve is faster, go ahead and lift the ordering code back into your mesh. On Fri, Oct 19, 2012 at 5:48 PM, Mohammad Mirzadeh wrote: > Thanks Barry. I think I agree with you that right way to do it is through > mesh setup. As a matter of fact I have done that for some of my problems. > Yet, I can think of situations where that kind of support might be > beneficial, especially if someone (a.k.a me in this case!) is looking for a > 'dirty and fast' fix. > > Anyway, I still believe that in long term, one should do the partitioning > in a pre-processing step anyways and so there may not be much incentive for > adding such a support. > > Thanks again, > Mohammad > > > On Fri, Oct 19, 2012 at 3:00 PM, Barry Smith wrote: > >> >> Mohammad, >> >> We've thought about adding this kind of support this over the years >> but always came to the conclusion that "the right way to do it" is during >> the mesh setup step. >> >> Barry >> >> >> On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh >> wrote: >> >> > Hi guys, >> > >> > Quick question. *After* the matrix is setup, is it possible to ask >> PETSc to renumber the nodes internally in the KSPSolve phase to minimize >> communications for MatMul inside the solver? Can I use the MatPartitioning >> object for this or is that only intended to partition the mesh as a >> pre-processing step? >> > >> > Thanks >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Fri Oct 19 18:03:44 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Fri, 19 Oct 2012 16:03:44 -0700 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: Cool. Jed, could you please point me to an example/tutorial for this? I'm going to give it a shot. Thanks On Fri, Oct 19, 2012 at 3:57 PM, Jed Brown wrote: > _You_ can compute a MatOrdering, MatPermute and VecPermute, solve, and > permute back. Profile and if the solve is faster, go ahead and lift the > ordering code back into your mesh. > > > On Fri, Oct 19, 2012 at 5:48 PM, Mohammad Mirzadeh wrote: > >> Thanks Barry. I think I agree with you that right way to do it is through >> mesh setup. As a matter of fact I have done that for some of my problems. >> Yet, I can think of situations where that kind of support might be >> beneficial, especially if someone (a.k.a me in this case!) is looking for a >> 'dirty and fast' fix. >> >> Anyway, I still believe that in long term, one should do the partitioning >> in a pre-processing step anyways and so there may not be much incentive for >> adding such a support. >> >> Thanks again, >> Mohammad >> >> >> On Fri, Oct 19, 2012 at 3:00 PM, Barry Smith wrote: >> >>> >>> Mohammad, >>> >>> We've thought about adding this kind of support this over the years >>> but always came to the conclusion that "the right way to do it" is during >>> the mesh setup step. >>> >>> Barry >>> >>> >>> On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh >>> wrote: >>> >>> > Hi guys, >>> > >>> > Quick question. *After* the matrix is setup, is it possible to ask >>> PETSc to renumber the nodes internally in the KSPSolve phase to minimize >>> communications for MatMul inside the solver? Can I use the MatPartitioning >>> object for this or is that only intended to partition the mesh as a >>> pre-processing step? >>> > >>> > Thanks >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stanshenbin at gmail.com Sat Oct 20 02:58:37 2012 From: stanshenbin at gmail.com (Bin Shen) Date: Sat, 20 Oct 2012 13:28:37 +0530 Subject: [petsc-users] Problem of installing PETS with MPI on windows 8 64 bit Message-ID: Hi, All, I would like to install PETS on Windows 8 with Cygwin. We get some error in the configuration and the configure.log is attached. With the following command ./configure --with-cc="win32fe cl" --with-fc=0 --download-c-blas-lapack=1 --with-mpi-include="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include" --with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/amd64/msmpi.lib" We get the following complain, while these two directories mentioned by the error messages are actually there, ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for detail s): ------------------------------------------------------------------------------- --with-mpi-lib=['/cygdrive/c/Program', 'Files/Microsoft', 'HPC', 'Pack', '2008', 'SDK/Lib/amd64/msmpi.lib'] and --with-mpi-include=['/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Inclu de'] did not work Any of you have some idea how to fix it? Thanks and best regards, Bin -- Bin Shen Dept. of Computer Science, Purdue University West Lafayette, IN., US http://www.cs.purdue.edu/homes/bshen/ -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 88 bytes Desc: not available URL: From knepley at gmail.com Sat Oct 20 08:33:01 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 20 Oct 2012 09:33:01 -0400 Subject: [petsc-users] [petsc-dev] Problem of installing PETS with MPI on windows 8 64 bit In-Reply-To: References: Message-ID: On Sat, Oct 20, 2012 at 3:58 AM, Bin Shen wrote: > Hi, All, > > I would like to install PETS on Windows 8 with Cygwin. We get some > error in the configuration and the configure.log is attached. > > With the following command > > ./configure --with-cc="win32fe cl" --with-fc=0 > --download-c-blas-lapack=1 --with-mpi-include="/cygdrive/c/Program > Files/Microsoft HPC Pack 2008 SDK/Include" > --with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 > SDK/Lib/amd64/msmpi.lib" > You need to send the actual log file ($PETSC_ARCH/conf/configure.log) to petsc-maint at mcs.anl.gov Matt > We get the following complain, while these two directories mentioned > by the error messages are actually there, > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > detail > s): > > ------------------------------------------------------------------------------- > --with-mpi-lib=['/cygdrive/c/Program', 'Files/Microsoft', 'HPC', 'Pack', > '2008', > 'SDK/Lib/amd64/msmpi.lib'] and > --with-mpi-include=['/cygdrive/c/Program Files/Microsoft HPC Pack 2008 > SDK/Inclu > de'] did not work > > > Any of you have some idea how to fix it? > > Thanks and best regards, > > Bin > > -- > Bin Shen > > Dept. of Computer Science, Purdue University > West Lafayette, IN., US > http://www.cs.purdue.edu/homes/bshen/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetan.jhurani at gmail.com Sat Oct 20 13:11:44 2012 From: chetan.jhurani at gmail.com (Chetan Jhurani) Date: Sat, 20 Oct 2012 11:11:44 -0700 Subject: [petsc-users] [petsc-dev] Problem of installing PETS with MPI on windows 8 64 bit In-Reply-To: References: Message-ID: <5082e963.c855420a.2496.0c5b@mx.google.com> Try --with-mpi-include="[/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include]" --with-mpi-lib="[/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/amd64/msmpi.lib]" Note the [] inside "". Chetan > -----Original Message----- > From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Bin Shen > Sent: Saturday, October 20, 2012 12:59 AM > To: petsc-users at mcs.anl.gov; petsc-dev at mcs.anl.gov > Subject: [petsc-dev] Problem of installing PETS with MPI on windows 8 64 bit > > Hi, All, > > I would like to install PETS on Windows 8 with Cygwin. We get some > error in the configuration and the configure.log is attached. > > With the following command > > ./configure --with-cc="win32fe cl" --with-fc=0 > --download-c-blas-lapack=1 --with-mpi-include="/cygdrive/c/Program > Files/Microsoft HPC Pack 2008 SDK/Include" > --with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 > SDK/Lib/amd64/msmpi.lib" > > We get the following complain, while these two directories mentioned > by the error messages are actually there, > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for detail > s): > ------------------------------------------------------------------------------- > --with-mpi-lib=['/cygdrive/c/Program', 'Files/Microsoft', 'HPC', 'Pack', '2008', > 'SDK/Lib/amd64/msmpi.lib'] and > --with-mpi-include=['/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Inclu > de'] did not work > > > Any of you have some idea how to fix it? > > Thanks and best regards, > > Bin > > -- > Bin Shen > > Dept. of Computer Science, Purdue University > West Lafayette, IN., US > http://www.cs.purdue.edu/homes/bshen/ From jedbrown at mcs.anl.gov Sat Oct 20 19:38:19 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 20 Oct 2012 19:38:19 -0500 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: On Fri, Oct 19, 2012 at 6:03 PM, Mohammad Mirzadeh wrote: > Cool. Jed, could you please point me to an example/tutorial for this? I'm > going to give it a shot. > This works for SeqAIJ: ierr = MatGetOrdering(A,ordering,&rowperm,&colperm);CHKERRQ(ierr); ierr = MatPermute(A,rowperm,colperm,&Aperm);CHKERRQ(ierr); ierr = VecPermute(b,colperm,PETSC_FALSE);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,Aperm,Aperm,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); ierr = VecPermute(x,rowperm,PETSC_TRUE);CHKERRQ(ierr); Now x is the solution back in the original ordering. There is an example in petsc-dev now. https://bitbucket.org/petsc/petsc-dev/src/tip/src/ksp/ksp/examples/tutorials/ex18.c It should work with petsc-3.3 in serial, but you need petsc-dev for parallel MatPermute. > > Thanks > > > On Fri, Oct 19, 2012 at 3:57 PM, Jed Brown wrote: > >> _You_ can compute a MatOrdering, MatPermute and VecPermute, solve, and >> permute back. Profile and if the solve is faster, go ahead and lift the >> ordering code back into your mesh. >> >> >> On Fri, Oct 19, 2012 at 5:48 PM, Mohammad Mirzadeh wrote: >> >>> Thanks Barry. I think I agree with you that right way to do it is >>> through mesh setup. As a matter of fact I have done that for some of my >>> problems. Yet, I can think of situations where that kind of support might >>> be beneficial, especially if someone (a.k.a me in this case!) is looking >>> for a 'dirty and fast' fix. >>> >>> Anyway, I still believe that in long term, one should do the >>> partitioning in a pre-processing step anyways and so there may not be much >>> incentive for adding such a support. >>> >>> Thanks again, >>> Mohammad >>> >>> >>> On Fri, Oct 19, 2012 at 3:00 PM, Barry Smith wrote: >>> >>>> >>>> Mohammad, >>>> >>>> We've thought about adding this kind of support this over the >>>> years but always came to the conclusion that "the right way to do it" is >>>> during the mesh setup step. >>>> >>>> Barry >>>> >>>> >>>> On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh >>>> wrote: >>>> >>>> > Hi guys, >>>> > >>>> > Quick question. *After* the matrix is setup, is it possible to ask >>>> PETSc to renumber the nodes internally in the KSPSolve phase to minimize >>>> communications for MatMul inside the solver? Can I use the MatPartitioning >>>> object for this or is that only intended to partition the mesh as a >>>> pre-processing step? >>>> > >>>> > Thanks >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mirzadeh at gmail.com Sat Oct 20 22:12:39 2012 From: mirzadeh at gmail.com (Mohammad Mirzadeh) Date: Sat, 20 Oct 2012 20:12:39 -0700 Subject: [petsc-users] matrix reordering In-Reply-To: References: Message-ID: Awesome. Thanks for doing this Jed. On Sat, Oct 20, 2012 at 5:38 PM, Jed Brown wrote: > On Fri, Oct 19, 2012 at 6:03 PM, Mohammad Mirzadeh wrote: > >> Cool. Jed, could you please point me to an example/tutorial for this? I'm >> going to give it a shot. >> > > This works for SeqAIJ: > > ierr = MatGetOrdering(A,ordering,&rowperm,&colperm);CHKERRQ(ierr); > ierr = MatPermute(A,rowperm,colperm,&Aperm);CHKERRQ(ierr); > ierr = VecPermute(b,colperm,PETSC_FALSE);CHKERRQ(ierr); > > ierr = > KSPSetOperators(ksp,Aperm,Aperm,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); > ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); > ierr = VecPermute(x,rowperm,PETSC_TRUE);CHKERRQ(ierr); > > Now x is the solution back in the original ordering. > > > There is an example in petsc-dev now. > > > https://bitbucket.org/petsc/petsc-dev/src/tip/src/ksp/ksp/examples/tutorials/ex18.c > > It should work with petsc-3.3 in serial, but you need petsc-dev for > parallel MatPermute. > > > >> >> Thanks >> >> >> On Fri, Oct 19, 2012 at 3:57 PM, Jed Brown wrote: >> >>> _You_ can compute a MatOrdering, MatPermute and VecPermute, solve, and >>> permute back. Profile and if the solve is faster, go ahead and lift the >>> ordering code back into your mesh. >>> >>> >>> On Fri, Oct 19, 2012 at 5:48 PM, Mohammad Mirzadeh wrote: >>> >>>> Thanks Barry. I think I agree with you that right way to do it is >>>> through mesh setup. As a matter of fact I have done that for some of my >>>> problems. Yet, I can think of situations where that kind of support might >>>> be beneficial, especially if someone (a.k.a me in this case!) is looking >>>> for a 'dirty and fast' fix. >>>> >>>> Anyway, I still believe that in long term, one should do the >>>> partitioning in a pre-processing step anyways and so there may not be much >>>> incentive for adding such a support. >>>> >>>> Thanks again, >>>> Mohammad >>>> >>>> >>>> On Fri, Oct 19, 2012 at 3:00 PM, Barry Smith wrote: >>>> >>>>> >>>>> Mohammad, >>>>> >>>>> We've thought about adding this kind of support this over the >>>>> years but always came to the conclusion that "the right way to do it" is >>>>> during the mesh setup step. >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Oct 19, 2012, at 4:16 PM, Mohammad Mirzadeh >>>>> wrote: >>>>> >>>>> > Hi guys, >>>>> > >>>>> > Quick question. *After* the matrix is setup, is it possible to ask >>>>> PETSc to renumber the nodes internally in the KSPSolve phase to minimize >>>>> communications for MatMul inside the solver? Can I use the MatPartitioning >>>>> object for this or is that only intended to partition the mesh as a >>>>> pre-processing step? >>>>> > >>>>> > Thanks >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sat Oct 20 22:55:46 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sat, 20 Oct 2012 22:55:46 -0500 Subject: [petsc-users] [petsc-maint #136411] PETSc Execution Error In-Reply-To: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> References: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> Message-ID: On Wed, Oct 17, 2012 at 1:45 PM, Barry Smith wrote: > > Paul, > > With this type of problem you need to get a handle on exactly where > the floating point exception appeared. This is done by running the debug > version of the code in a debugger and asking the debugger to catch floating > point exceptions (how to do this depends on the debugger). Once you see > exactly where the floating point except ion comes from (for example a > divide by zero, square root of a negative number etc) that can lead to > seeing how to avoid it. > > Note to developers, we should have an answer for this in PETSc FAQ. > https://bitbucket.org/petsc/petsc-dev/changeset/6e2912899260e1c94b78c60754198048af02ca85 Can one of you Mac users figure out how to make -fp_trap work there? -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin.kirk-1 at nasa.gov Sun Oct 21 07:17:04 2012 From: benjamin.kirk-1 at nasa.gov (Kirk, Benjamin (JSC-EG311)) Date: Sun, 21 Oct 2012 07:17:04 -0500 Subject: [petsc-users] [petsc-maint #136411] PETSc Execution Error In-Reply-To: References: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> Message-ID: <39E865E9-D7C4-44D7-AF85-0424A9A8E7A3@nasa.gov> On Oct 20, 2012, at 10:55 PM, Jed Brown wrote: > On Wed, Oct 17, 2012 at 1:45 PM, Barry Smith wrote: > > Paul, > > With this type of problem you need to get a handle on exactly where the floating point exception appeared. This is done by running the debug version of the code in a debugger and asking the debugger to catch floating point exceptions (how to do this depends on the debugger). Once you see exactly where the floating point except ion comes from (for example a divide by zero, square root of a negative number etc) that can lead to seeing how to avoid it. > > Note to developers, we should have an answer for this in PETSc FAQ. > > https://bitbucket.org/petsc/petsc-dev/changeset/6e2912899260e1c94b78c60754198048af02ca85 > > Can one of you Mac users figure out how to make -fp_trap work there? Aside from the debugger, you can enable floating point exceptions at runtime if there is interest. Below is the code we use on mac/linux in libMesh to honor --enable-fpe on the command line. Note this was adapted from code originally provided from INL by the Moose team, so thanks really goes to them! We prefer the feenableexcept() API on linux, but on the Mac you can get access to most of the exceptions through the MMX instruction set. Independent of the debugger it will still print the actual error and a stack trace, which is often good enough to know what happened. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // floating-point exceptions #ifdef LIBMESH_HAVE_FENV_H # include #endif #ifdef LIBMESH_HAVE_XMMINTRIN_H # include #endif /** * Floating point exception handler -- courtesy of Cody Permann & MOOSE team */ void libmesh_handleFPE(int /*signo*/, siginfo_t *info, void * /*context*/) { std::cout << std::endl; std::cout << "Floating point exception signaled ("; switch (info->si_code) { case FPE_INTDIV: std::cerr << "integer divide by zero"; break; case FPE_INTOVF: std::cerr << "integer overflow"; break; case FPE_FLTDIV: std::cerr << "floating point divide by zero"; break; case FPE_FLTOVF: std::cerr << "floating point overflow"; break; case FPE_FLTUND: std::cerr << "floating point underflow"; break; case FPE_FLTRES: std::cerr << "floating point inexact result"; break; case FPE_FLTINV: std::cerr << "invalid floating point operation"; break; case FPE_FLTSUB: std::cerr << "subscript out of range"; break; } std::cout << ")!" << std::endl; std::cout << std::endl; std::cout << "To track this down, compile debug version, start debugger, set breakpoint for 'libmesh_handleFPE' and run" << std::endl; std::cout << "In gdb do:" << std::endl; std::cout << " break libmesh_handleFPE" << std::endl; std::cout << " run ..." << std::endl; std::cout << " bt" << std::endl; libmesh_error(); } /** * Toggle floating point exceptions -- courtesy of Cody Permann & MOOSE team */ void enableFPE(bool on) { #if !defined(LIBMESH_HAVE_FEENABLEEXCEPT) && defined(LIBMESH_HAVE_XMMINTRIN_H) static int flags = 0; #endif if (on) { struct sigaction new_action, old_action; #ifdef LIBMESH_HAVE_FEENABLEEXCEPT feenableexcept(FE_DIVBYZERO | FE_INVALID); #elif LIBMESH_HAVE_XMMINTRIN_H flags = _MM_GET_EXCEPTION_MASK(); // store the flags _MM_SET_EXCEPTION_MASK(flags & ~_MM_MASK_INVALID); #endif // Set up the structure to specify the new action. new_action.sa_sigaction = libmesh_handleFPE; sigemptyset (&new_action.sa_mask); new_action.sa_flags = SA_SIGINFO; sigaction (SIGFPE, NULL, &old_action); if (old_action.sa_handler != SIG_IGN) sigaction (SIGFPE, &new_action, NULL); } else { #ifdef LIBMESH_HAVE_FEDISABLEEXCEPT fedisableexcept(FE_DIVBYZERO | FE_INVALID); #elif LIBMESH_HAVE_XMMINTRIN_H _MM_SET_EXCEPTION_MASK(flags); #endif signal(SIGFPE, 0); } } From jedbrown at mcs.anl.gov Sun Oct 21 09:01:30 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 21 Oct 2012 09:01:30 -0500 Subject: [petsc-users] [petsc-maint #136411] PETSc Execution Error In-Reply-To: <39E865E9-D7C4-44D7-AF85-0424A9A8E7A3@nasa.gov> References: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> <39E865E9-D7C4-44D7-AF85-0424A9A8E7A3@nasa.gov> Message-ID: On Sun, Oct 21, 2012 at 7:17 AM, Kirk, Benjamin (JSC-EG311) < benjamin.kirk-1 at nasa.gov> wrote: > Aside from the debugger, you can enable floating point exceptions at > runtime if there is interest. > Thanks for this. There is code for Linux (using feenableexcept) and a bunch of esoteric architectures in petsc/src/sys/error/fp.c, activated when you pass -fp_trap. It looks like the xmmintrin option there wasn't correct. Also, I see that the specific cause of the exception is actually retained in si_code, but with a different set of macros (FPE_FLTDIV) than feenableexcept (FE_DIVBYZERO). Satish and Barry, is there any rationale for still using signal(2), which has been deprecated since 1988, instead of sigaction()? > > Below is the code we use on mac/linux in libMesh to honor --enable-fpe on > the command line. Note this was adapted from code originally provided from > INL by the Moose team, so thanks really goes to them! We prefer the > feenableexcept() API on linux, but on the Mac you can get access to most of > the exceptions through the MMX instruction set. > > Independent of the debugger it will still print the actual error and a > stack trace, which is often good enough to know what happened. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > // floating-point exceptions > #ifdef LIBMESH_HAVE_FENV_H > # include > #endif > #ifdef LIBMESH_HAVE_XMMINTRIN_H > # include > #endif > > > > /** > * Floating point exception handler -- courtesy of Cody Permann & MOOSE > team > */ > void libmesh_handleFPE(int /*signo*/, siginfo_t *info, void * > /*context*/) > { > std::cout << std::endl; > std::cout << "Floating point exception signaled ("; > switch (info->si_code) > { > case FPE_INTDIV: std::cerr << "integer divide by zero"; break; > case FPE_INTOVF: std::cerr << "integer overflow"; break; > case FPE_FLTDIV: std::cerr << "floating point divide by zero"; break; > case FPE_FLTOVF: std::cerr << "floating point overflow"; break; > case FPE_FLTUND: std::cerr << "floating point underflow"; break; > case FPE_FLTRES: std::cerr << "floating point inexact result"; break; > case FPE_FLTINV: std::cerr << "invalid floating point operation"; > break; > case FPE_FLTSUB: std::cerr << "subscript out of range"; break; > } > std::cout << ")!" << std::endl; > > std::cout << std::endl; > std::cout << "To track this down, compile debug version, start > debugger, set breakpoint for 'libmesh_handleFPE' and run" << std::endl; > std::cout << "In gdb do:" << std::endl; > std::cout << " break libmesh_handleFPE" << std::endl; > std::cout << " run ..." << std::endl; > std::cout << " bt" << std::endl; > > libmesh_error(); > } > > > /** > * Toggle floating point exceptions -- courtesy of Cody Permann & MOOSE > team > */ > void enableFPE(bool on) > { > #if !defined(LIBMESH_HAVE_FEENABLEEXCEPT) && > defined(LIBMESH_HAVE_XMMINTRIN_H) > static int flags = 0; > #endif > > if (on) > { > struct sigaction new_action, old_action; > > #ifdef LIBMESH_HAVE_FEENABLEEXCEPT > feenableexcept(FE_DIVBYZERO | FE_INVALID); > #elif LIBMESH_HAVE_XMMINTRIN_H > flags = _MM_GET_EXCEPTION_MASK(); // store the flags > _MM_SET_EXCEPTION_MASK(flags & ~_MM_MASK_INVALID); > #endif > > > // Set up the structure to specify the new action. > new_action.sa_sigaction = libmesh_handleFPE; > sigemptyset (&new_action.sa_mask); > new_action.sa_flags = SA_SIGINFO; > > sigaction (SIGFPE, NULL, &old_action); > if (old_action.sa_handler != SIG_IGN) > sigaction (SIGFPE, &new_action, NULL); > } > else > { > #ifdef LIBMESH_HAVE_FEDISABLEEXCEPT > fedisableexcept(FE_DIVBYZERO | FE_INVALID); > #elif LIBMESH_HAVE_XMMINTRIN_H > _MM_SET_EXCEPTION_MASK(flags); > #endif > signal(SIGFPE, 0); > } > } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sun Oct 21 11:15:30 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sun, 21 Oct 2012 11:15:30 -0500 Subject: [petsc-users] [petsc-maint #136411] PETSc Execution Error In-Reply-To: References: <8ECB7CEB-EE87-4975-B47A-30A34BC0DC39@mcs.anl.gov> <39E865E9-D7C4-44D7-AF85-0424A9A8E7A3@nasa.gov> Message-ID: On Oct 21, 2012, at 9:01 AM, Jed Brown wrote: > On Sun, Oct 21, 2012 at 7:17 AM, Kirk, Benjamin (JSC-EG311) < > benjamin.kirk-1 at nasa.gov> wrote: > >> Aside from the debugger, you can enable floating point exceptions at >> runtime if there is interest. >> > > Thanks for this. There is code for Linux (using feenableexcept) and a bunch > of esoteric architectures in petsc/src/sys/error/fp.c, activated when you > pass -fp_trap. It looks like the xmmintrin option there wasn't correct. > > Also, I see that the specific cause of the exception is actually retained > in si_code, but with a different set of macros (FPE_FLTDIV) than > feenableexcept (FE_DIVBYZERO). > > Satish and Barry, is there any rationale for still using signal(2), which > has been deprecated since 1988, instead of sigaction()? I think just 1) inertia and 2) if it ain't broke don't fix it (and break it in the process). Presumably this could go on your always growing, never shrinking, projects list on bitbucket. Make sure to check what would happen with Microsoft Windows before making a change. Barry > > >> >> Below is the code we use on mac/linux in libMesh to honor --enable-fpe on >> the command line. Note this was adapted from code originally provided from >> INL by the Moose team, so thanks really goes to them! We prefer the >> feenableexcept() API on linux, but on the Mac you can get access to most of >> the exceptions through the MMX instruction set. >> >> Independent of the debugger it will still print the actual error and a >> stack trace, which is often good enough to know what happened. >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> // floating-point exceptions >> #ifdef LIBMESH_HAVE_FENV_H >> # include >> #endif >> #ifdef LIBMESH_HAVE_XMMINTRIN_H >> # include >> #endif >> >> >> >> /** >> * Floating point exception handler -- courtesy of Cody Permann & MOOSE >> team >> */ >> void libmesh_handleFPE(int /*signo*/, siginfo_t *info, void * >> /*context*/) >> { >> std::cout << std::endl; >> std::cout << "Floating point exception signaled ("; >> switch (info->si_code) >> { >> case FPE_INTDIV: std::cerr << "integer divide by zero"; break; >> case FPE_INTOVF: std::cerr << "integer overflow"; break; >> case FPE_FLTDIV: std::cerr << "floating point divide by zero"; break; >> case FPE_FLTOVF: std::cerr << "floating point overflow"; break; >> case FPE_FLTUND: std::cerr << "floating point underflow"; break; >> case FPE_FLTRES: std::cerr << "floating point inexact result"; break; >> case FPE_FLTINV: std::cerr << "invalid floating point operation"; >> break; >> case FPE_FLTSUB: std::cerr << "subscript out of range"; break; >> } >> std::cout << ")!" << std::endl; >> >> std::cout << std::endl; >> std::cout << "To track this down, compile debug version, start >> debugger, set breakpoint for 'libmesh_handleFPE' and run" << std::endl; >> std::cout << "In gdb do:" << std::endl; >> std::cout << " break libmesh_handleFPE" << std::endl; >> std::cout << " run ..." << std::endl; >> std::cout << " bt" << std::endl; >> >> libmesh_error(); >> } >> >> >> /** >> * Toggle floating point exceptions -- courtesy of Cody Permann & MOOSE >> team >> */ >> void enableFPE(bool on) >> { >> #if !defined(LIBMESH_HAVE_FEENABLEEXCEPT) && >> defined(LIBMESH_HAVE_XMMINTRIN_H) >> static int flags = 0; >> #endif >> >> if (on) >> { >> struct sigaction new_action, old_action; >> >> #ifdef LIBMESH_HAVE_FEENABLEEXCEPT >> feenableexcept(FE_DIVBYZERO | FE_INVALID); >> #elif LIBMESH_HAVE_XMMINTRIN_H >> flags = _MM_GET_EXCEPTION_MASK(); // store the flags >> _MM_SET_EXCEPTION_MASK(flags & ~_MM_MASK_INVALID); >> #endif >> >> >> // Set up the structure to specify the new action. >> new_action.sa_sigaction = libmesh_handleFPE; >> sigemptyset (&new_action.sa_mask); >> new_action.sa_flags = SA_SIGINFO; >> >> sigaction (SIGFPE, NULL, &old_action); >> if (old_action.sa_handler != SIG_IGN) >> sigaction (SIGFPE, &new_action, NULL); >> } >> else >> { >> #ifdef LIBMESH_HAVE_FEDISABLEEXCEPT >> fedisableexcept(FE_DIVBYZERO | FE_INVALID); >> #elif LIBMESH_HAVE_XMMINTRIN_H >> _MM_SET_EXCEPTION_MASK(flags); >> #endif >> signal(SIGFPE, 0); >> } >> } >> > From zhenglun.wei at gmail.com Mon Oct 22 11:14:48 2012 From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei) Date: Mon, 22 Oct 2012 11:14:48 -0500 Subject: [petsc-users] Need a diagnose for the slow performance Message-ID: <508570F8.2020406@gmail.com> Dear all, I hope you're having a nice day. I need some help to figure out what is wrong in my code. I had very good performance after I change the preconditioner. The out_1, which is attached, shows that. Last week, in order to use PETSc in out new cluster, I reconfigure the PETSc with the same configuration in the new cluster and saved to another directory. My code can obtain very good performance in the new machine. However, my code performs very bad in the old cluster, as shown in out_2 in the attachment. Does any one have any idea on this? thanks, Alan -------------- next part -------------- NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 NumRank = 48 node = 11 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 11 xs = 114 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 12 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 12 xs = 152 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 28 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 28 xs = 152 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 36 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 36 xs = 152 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 44 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 44 xs = 152 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 43 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 43 xs = 114 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 27 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 27 xs = 114 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 19 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 19 xs = 114 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 3 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 3 xs = 114 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 35 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 35 xs = 114 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 2 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 2 xs = 76 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 0 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 0 xs = 0 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 18 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 18 xs = 76 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 10 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 10 xs = 76 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 8 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 8 xs = 0 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 32 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 32 xs = 0 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 33 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 33 xs = 38 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 1 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 1 xs = 38 ys = 0 zs = 0 xw = 38 yw = 48 zw = 32 node = 16 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 16 xs = 0 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 24 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 24 xs = 0 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 40 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 34 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 34 xs = 76 ys = 0 zs = 64 xw = 38 yw = 48 zw = 32 node = 26 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 26 xs = 76 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 40 xs = 0 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 42 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 42 xs = 76 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 13 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 13 xs = 189 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 21 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 21 xs = 189 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 45 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 45 xs = 189 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 37 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 37 xs = 189 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 38 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 38 xs = 226 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 39 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 39 xs = 263 ys = 0 zs = 64 xw = 37 yw = 48 zw = 32 node = 47 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 47 xs = 263 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 node = 25 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 25 xs = 38 ys = 48 zs = 32 xw = 38 yw = 48 zw = 32 node = 41 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 41 xs = 38 ys = 48 zs = 64 xw = 38 yw = 48 zw = 32 node = 17 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 17 xs = 38 ys = 0 zs = 32 xw = 38 yw = 48 zw = 32 node = 29 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 29 xs = 189 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 31 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 31 xs = 263 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 9 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 9 xs = 38 ys = 48 zs = 0 xw = 38 yw = 48 zw = 32 node = 20 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 20 xs = 152 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 15 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 15 xs = 263 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 23 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 23 xs = 263 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 4 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 4 xs = 152 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 6 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 6 xs = 226 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 7 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 7 xs = 263 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 22 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 22 xs = 226 ys = 0 zs = 32 xw = 37 yw = 48 zw = 32 node = 30 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 30 xs = 226 ys = 48 zs = 32 xw = 37 yw = 48 zw = 32 node = 5 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 5 xs = 189 ys = 0 zs = 0 xw = 37 yw = 48 zw = 32 node = 14 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 14 xs = 226 ys = 48 zs = 0 xw = 37 yw = 48 zw = 32 node = 46 mx = 300 my = 96 mz = 96 mm = 8 nn = 2 pp = 3 node = 46 xs = 226 ys = 48 zs = 64 xw = 37 yw = 48 zw = 32 rank 18 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 35 front: 59 rank = 18, NbrRanks are: L: 17, R: 19, B:10, T:26, A:2, F:34!! rank = 18: xs = 76, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 18: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 18: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 10 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 35 front: 32 rank = 10, NbrRanks are: L: 9, R: 11, B:2, T:18, A:-6, F:26!! rank = 10: xs = 76, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 10: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 10: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 46 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank 9 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 35 front: 32 rank = 46, NbrRanks are: L: 45, R: 47, B:38, T:54, A:30, F:62!! rank = 9, NbrRanks are: L: 8, R: 10, B:1, T:17, A:-7, F:25!! rank = 46: xs = 226, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 9: xs = 38, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 46: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 9: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 46: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 9: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 11 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 11, NbrRanks are: L: 10, R: 12, B:3, T:19, A:-5, F:27!! rank = 11: xs = 114, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 11: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 11: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 45 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 45, NbrRanks are: L: 44, R: 46, B:37, T:53, A:29, F:61!! rank 6 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 6, NbrRanks are: L: 5, R: 7, B:-2, T:14, A:-10, F:22!! rank = 6: xs = 226, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 6: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 6: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 45: xs = 189, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 45: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 45: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 47 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank 8 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 35 front: 32 rank = 8, NbrRanks are: L: 7, R: 9, B:0, T:16, A:-8, F:24!! rank = 8: xs = 0, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 8: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 8: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 44 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 44, NbrRanks are: L: 43, R: 45, B:36, T:52, A:28, F:60!! rank = 44: xs = 152, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 44: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 44: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 47, NbrRanks are: L: 46, R: 48, B:39, T:55, A:31, F:63!! rank = 47: xs = 263, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 47: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 47: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 16 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 35 front: 59 rank = 16, NbrRanks are: L: 15, R: 17, B:8, T:24, A:0, F:32!! rank = 16: xs = 0, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 16: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 16: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 17 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 35 front: 59 rank = 17, NbrRanks are: L: 16, R: 18, B:9, T:25, A:1, F:33!! rank = 17: xs = 38, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 17: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 17: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 19 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 19, NbrRanks are: L: 18, R: 20, B:11, T:27, A:3, F:35!! rank = 19: xs = 114, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 19: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 19: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 4 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 4, NbrRanks are: L: 3, R: 5, B:-4, T:12, A:-12, F:20!! rank = 4: xs = 152, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 4: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 4: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 5 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 5, NbrRanks are: L: 4, R: 6, B:-3, T:13, A:-11, F:21!! rank = 5: xs = 189, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 5: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 5: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 7 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 7, NbrRanks are: L: 6, R: 8, B:-1, T:15, A:-9, F:23!! rank = 7: xs = 263, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 7: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 7: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 38 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 38, NbrRanks are: L: 37, R: 39, B:30, T:46, A:22, F:54!! rank = 38: xs = 226, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 38: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 38: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 14 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 14, NbrRanks are: L: 13, R: 15, B:6, T:22, A:-2, F:30!! rank = 14: xs = 226, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 14: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 14: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 22 Cylinder: neighbor = left: 226 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 22, NbrRanks are: L: 21, R: 23, B:14, T:30, A:6, F:38!! rank = 22: xs = 226, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 22: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 22: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 37 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 37, NbrRanks are: L: 36, R: 38, B:29, T:45, A:21, F:53!! rank = 37: xs = 189, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank 13 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank 21 Cylinder: neighbor = left: 189 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank 0 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 35 front: 32 rank = 0, NbrRanks are: L: -1, R: 1, B:-8, T:8, A:-16, F:16!! rank = 0: xs = 0, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 0: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 0: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank = 37: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 37: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 13, NbrRanks are: L: 12, R: 14, B:5, T:21, A:-3, F:29!! rank = 13: xs = 189, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 13: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 21, NbrRanks are: L: 20, R: 22, B:13, T:29, A:5, F:37!! rank = 21: xs = 189, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 21: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank 2 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 35 front: 32 rank = 2, NbrRanks are: L: 1, R: 3, B:-6, T:10, A:-14, F:18!! rank = 2: xs = 76, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 2: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 2: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank = 13: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 21: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 3 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 35 front: 32 rank = 3, NbrRanks are: L: 2, R: 4, B:-5, T:11, A:-13, F:19!! rank = 3: xs = 114, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 3: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 3: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 20 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 20, NbrRanks are: L: 19, R: 21, B:12, T:28, A:4, F:36!! rank = 20: xs = 152, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 20: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 20: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 12 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 12, NbrRanks are: L: 11, R: 13, B:4, T:20, A:-4, F:28!! rank = 12: xs = 152, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank 15 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 35 front: 32 rank = 15, NbrRanks are: L: 14, R: 16, B:7, T:23, A:-1, F:31!! rank = 15: xs = 263, ys = 48, zs = 0, mx = 300, my = 96, mz = 96! rank = 12: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 12: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank = 15: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 15: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 24 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 35 front: 59 rank = 24, NbrRanks are: L: 23, R: 25, B:16, T:32, A:8, F:40!! rank = 24: xs = 0, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 24: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 24: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 36 Cylinder: neighbor = left: 152 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 36, NbrRanks are: L: 35, R: 37, B:28, T:44, A:20, F:52!! rank = 36: xs = 152, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 36: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 36: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 40 Cylinder: neighbor = left: 108 right: 38 bottom: 48 top: 59 aft: 64 front: 59 rank = 40, NbrRanks are: L: 39, R: 41, B:32, T:48, A:24, F:56!! rank = 40: xs = 0, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 40: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 40: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 28 Cylinder: neighbor = left: 152 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 28, NbrRanks are: L: 27, R: 29, B:20, T:36, A:12, F:44!! rank = 28: xs = 152, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 28: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 28: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 23 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 35 front: 59 rank = 23, NbrRanks are: L: 22, R: 24, B:15, T:31, A:7, F:39!! rank = 23: xs = 263, ys = 0, zs = 32, mx = 300, my = 96, mz = 96! rank = 23: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 23: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 25 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 35 front: 59 rank = 25, NbrRanks are: L: 24, R: 26, B:17, T:33, A:9, F:41!! rank = 25: xs = 38, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 25: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 25: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 39 Cylinder: neighbor = left: 263 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 39, NbrRanks are: L: 38, R: 40, B:31, T:47, A:23, F:55!! rank = 39: xs = 263, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 39: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 39: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 41 Cylinder: neighbor = left: 108 right: 76 bottom: 48 top: 59 aft: 64 front: 59 rank = 41, NbrRanks are: L: 40, R: 42, B:33, T:49, A:25, F:57!! rank = 41: xs = 38, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 41: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 41: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 29 Cylinder: neighbor = left: 189 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 29, NbrRanks are: L: 28, R: 30, B:21, T:37, A:13, F:45!! rank = 29: xs = 189, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 29: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 29: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 26 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 35 front: 59 rank = 26, NbrRanks are: L: 25, R: 27, B:18, T:34, A:10, F:42!! rank = 26: xs = 76, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 26: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 26: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 42 Cylinder: neighbor = left: 108 right: 114 bottom: 48 top: 59 aft: 64 front: 59 rank = 42, NbrRanks are: L: 41, R: 43, B:34, T:50, A:26, F:58!! rank = 42: xs = 76, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 42: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 42: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 30 Cylinder: neighbor = left: 226 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 30, NbrRanks are: L: 29, R: 31, B:22, T:38, A:14, F:46!! rank = 30: xs = 226, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 30: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 30: xc = 37, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 27 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 27, NbrRanks are: L: 26, R: 28, B:19, T:35, A:11, F:43!! rank = 27: xs = 114, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 27: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 27: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 43 Cylinder: neighbor = left: 114 right: 132 bottom: 48 top: 59 aft: 64 front: 59 rank = 43, NbrRanks are: L: 42, R: 44, B:35, T:51, A:27, F:59!! rank = 43: xs = 114, ys = 48, zs = 64, mx = 300, my = 96, mz = 96! rank = 43: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 43: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 31 Cylinder: neighbor = left: 263 right: 132 bottom: 48 top: 59 aft: 35 front: 59 rank = 31, NbrRanks are: L: 30, R: 32, B:23, T:39, A:15, F:47!! rank = 31: xs = 263, ys = 48, zs = 32, mx = 300, my = 96, mz = 96! rank = 31: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 31: xc = 36, yc = 48, zc = 32, xw = 37, yw = 48, zw = 32 rank 1 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 35 front: 32 rank = 1, NbrRanks are: L: 0, R: 2, B:-7, T:9, A:-15, F:17!! rank = 1: xs = 38, ys = 0, zs = 0, mx = 300, my = 96, mz = 96! rank = 1: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 1: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 32 Cylinder: neighbor = left: 108 right: 38 bottom: 35 top: 48 aft: 64 front: 59 rank = 32, NbrRanks are: L: 31, R: 33, B:24, T:40, A:16, F:48!! rank = 32: xs = 0, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 32: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 32: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 33 Cylinder: neighbor = left: 108 right: 76 bottom: 35 top: 48 aft: 64 front: 59 rank = 33, NbrRanks are: L: 32, R: 34, B:25, T:41, A:17, F:49!! rank = 33: xs = 38, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 33: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 33: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 34 Cylinder: neighbor = left: 108 right: 114 bottom: 35 top: 48 aft: 64 front: 59 rank = 34, NbrRanks are: L: 33, R: 35, B:26, T:42, A:18, F:50!! rank = 34: xs = 76, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 34: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 34: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 rank 35 Cylinder: neighbor = left: 114 right: 132 bottom: 35 top: 48 aft: 64 front: 59 rank = 35, NbrRanks are: L: 34, R: 36, B:27, T:43, A:19, F:51!! rank = 35: xs = 114, ys = 0, zs = 64, mx = 300, my = 96, mz = 96! rank = 35: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 35: xc = 38, yc = 48, zc = 32, xw = 38, yw = 48, zw = 32 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 Current time step = 0, Total time step number = 5 0 KSP unpreconditioned resid norm 2.823135537733e+00 true resid norm 2.823135537733e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 2.154307416984e-01 true resid norm 2.154307416984e-01 ||r(i)||/||b|| 7.630903257002e-02 2 KSP unpreconditioned resid norm 6.616308524617e-02 true resid norm 6.616308524617e-02 ||r(i)||/||b|| 2.343602861494e-02 3 KSP unpreconditioned resid norm 3.244770285497e-02 true resid norm 3.244770285497e-02 ||r(i)||/||b|| 1.149349807024e-02 4 KSP unpreconditioned resid norm 2.037399364799e-02 true resid norm 2.037399364799e-02 ||r(i)||/||b|| 7.216796138789e-03 5 KSP unpreconditioned resid norm 1.115303682743e-02 true resid norm 1.115303682743e-02 ||r(i)||/||b|| 3.950584971341e-03 6 KSP unpreconditioned resid norm 7.349490089262e-03 true resid norm 7.349490089262e-03 ||r(i)||/||b|| 2.603307560346e-03 7 KSP unpreconditioned resid norm 5.032355046504e-03 true resid norm 5.032355046504e-03 ||r(i)||/||b|| 1.782541071530e-03 8 KSP unpreconditioned resid norm 3.412030504773e-03 true resid norm 3.412030504773e-03 ||r(i)||/||b|| 1.208596066030e-03 9 KSP unpreconditioned resid norm 2.506624879726e-03 true resid norm 2.506624879726e-03 ||r(i)||/||b|| 8.878868358331e-04 10 KSP unpreconditioned resid norm 1.911583249761e-03 true resid norm 1.911583249761e-03 ||r(i)||/||b|| 6.771135229646e-04 11 KSP unpreconditioned resid norm 1.450523741360e-03 true resid norm 1.450523741359e-03 ||r(i)||/||b|| 5.137988318210e-04 12 KSP unpreconditioned resid norm 1.165820441830e-03 true resid norm 1.165820441830e-03 ||r(i)||/||b|| 4.129523454500e-04 13 KSP unpreconditioned resid norm 9.075966183811e-04 true resid norm 9.075966183811e-04 ||r(i)||/||b|| 3.214853152640e-04 14 KSP unpreconditioned resid norm 7.190731823975e-04 true resid norm 7.190731823977e-04 ||r(i)||/||b|| 2.547072830145e-04 15 KSP unpreconditioned resid norm 5.738556352769e-04 true resid norm 5.738556352768e-04 ||r(i)||/||b|| 2.032688929054e-04 16 KSP unpreconditioned resid norm 4.542858026836e-04 true resid norm 4.542858026837e-04 ||r(i)||/||b|| 1.609153356656e-04 17 KSP unpreconditioned resid norm 3.691546556208e-04 true resid norm 3.691546556209e-04 ||r(i)||/||b|| 1.307605145722e-04 18 KSP unpreconditioned resid norm 3.023816670689e-04 true resid norm 3.023816670686e-04 ||r(i)||/||b|| 1.071084483997e-04 19 KSP unpreconditioned resid norm 2.446493662857e-04 true resid norm 2.446493662856e-04 ||r(i)||/||b|| 8.665873919820e-05 20 KSP unpreconditioned resid norm 2.063235929689e-04 true resid norm 2.063235929687e-04 ||r(i)||/||b|| 7.308313405824e-05 21 KSP unpreconditioned resid norm 1.690424749200e-04 true resid norm 1.690424749199e-04 ||r(i)||/||b|| 5.987756261099e-05 22 KSP unpreconditioned resid norm 1.401907850897e-04 true resid norm 1.401907850896e-04 ||r(i)||/||b|| 4.965783017354e-05 23 KSP unpreconditioned resid norm 1.144389156826e-04 true resid norm 1.144389156828e-04 ||r(i)||/||b|| 4.053610397137e-05 24 KSP unpreconditioned resid norm 8.920795008676e-05 true resid norm 8.920795008653e-05 ||r(i)||/||b|| 3.159889027438e-05 25 KSP unpreconditioned resid norm 6.909908488215e-05 true resid norm 6.909908488195e-05 ||r(i)||/||b|| 2.447600689319e-05 26 KSP unpreconditioned resid norm 5.266847004343e-05 true resid norm 5.266847004323e-05 ||r(i)||/||b|| 1.865601893330e-05 27 KSP unpreconditioned resid norm 4.064559966622e-05 true resid norm 4.064559966603e-05 ||r(i)||/||b|| 1.439732493278e-05 28 KSP unpreconditioned resid norm 3.171947373934e-05 true resid norm 3.171947373916e-05 ||r(i)||/||b|| 1.123554760840e-05 29 KSP unpreconditioned resid norm 2.462933869897e-05 true resid norm 2.462933869893e-05 ||r(i)||/||b|| 8.724107776529e-06 30 KSP unpreconditioned resid norm 1.822375473757e-05 true resid norm 1.822375473757e-05 ||r(i)||/||b|| 6.455146943529e-06 31 KSP unpreconditioned resid norm 1.574617811541e-05 true resid norm 1.574617811542e-05 ||r(i)||/||b|| 5.577549467591e-06 32 KSP unpreconditioned resid norm 1.307785144339e-05 true resid norm 1.307785144340e-05 ||r(i)||/||b|| 4.632385256962e-06 33 KSP unpreconditioned resid norm 1.142770280984e-05 true resid norm 1.142770280984e-05 ||r(i)||/||b|| 4.047876078603e-06 34 KSP unpreconditioned resid norm 9.171371046918e-06 true resid norm 9.171371046933e-06 ||r(i)||/||b|| 3.248647089150e-06 35 KSP unpreconditioned resid norm 7.576417541607e-06 true resid norm 7.576417541619e-06 ||r(i)||/||b|| 2.683688912684e-06 36 KSP unpreconditioned resid norm 6.341122445581e-06 true resid norm 6.341122445582e-06 ||r(i)||/||b|| 2.246127527648e-06 37 KSP unpreconditioned resid norm 5.520660583479e-06 true resid norm 5.520660583471e-06 ||r(i)||/||b|| 1.955506744074e-06 38 KSP unpreconditioned resid norm 4.758050788581e-06 true resid norm 4.758050788571e-06 ||r(i)||/||b|| 1.685378092896e-06 39 KSP unpreconditioned resid norm 4.119934417329e-06 true resid norm 4.119934417327e-06 ||r(i)||/||b|| 1.459347014078e-06 40 KSP unpreconditioned resid norm 3.423330512164e-06 true resid norm 3.423330512170e-06 ||r(i)||/||b|| 1.212598710340e-06 41 KSP unpreconditioned resid norm 2.843937477031e-06 true resid norm 2.843937477043e-06 ||r(i)||/||b|| 1.007368381373e-06 42 KSP unpreconditioned resid norm 2.247136771553e-06 true resid norm 2.247136771556e-06 ||r(i)||/||b|| 7.959719756711e-07 43 KSP unpreconditioned resid norm 1.767055534296e-06 true resid norm 1.767055534289e-06 ||r(i)||/||b|| 6.259194823171e-07 44 KSP unpreconditioned resid norm 1.330561337297e-06 true resid norm 1.330561337289e-06 ||r(i)||/||b|| 4.713062194518e-07 45 KSP unpreconditioned resid norm 1.002587660836e-06 true resid norm 1.002587660849e-06 ||r(i)||/||b|| 3.551326698448e-07 46 KSP unpreconditioned resid norm 7.571869187267e-07 true resid norm 7.571869187577e-07 ||r(i)||/||b|| 2.682077812551e-07 47 KSP unpreconditioned resid norm 5.557160212082e-07 true resid norm 5.557160212518e-07 ||r(i)||/||b|| 1.968435499551e-07 48 KSP unpreconditioned resid norm 4.235338619283e-07 true resid norm 4.235338619715e-07 ||r(i)||/||b|| 1.500225038121e-07 49 KSP unpreconditioned resid norm 3.213108718199e-07 true resid norm 3.213108718685e-07 ||r(i)||/||b|| 1.138134770981e-07 50 KSP unpreconditioned resid norm 2.465819253483e-07 true resid norm 2.465819253946e-07 ||r(i)||/||b|| 8.734328270778e-08 Linear solve converged due to CONVERGED_RTOL iterations 50 Residual norm 2.46582e-07 0 KSP unpreconditioned resid norm 2.464333785299e+00 true resid norm 2.464333785299e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.992196609304e-01 true resid norm 1.992196609304e-01 ||r(i)||/||b|| 8.084118398199e-02 2 KSP unpreconditioned resid norm 6.477029955211e-02 true resid norm 6.477029955211e-02 ||r(i)||/||b|| 2.628308711202e-02 3 KSP unpreconditioned resid norm 3.257461563671e-02 true resid norm 3.257461563671e-02 ||r(i)||/||b|| 1.321842675332e-02 4 KSP unpreconditioned resid norm 2.205871203061e-02 true resid norm 2.205871203061e-02 ||r(i)||/||b|| 8.951186792227e-03 5 KSP unpreconditioned resid norm 1.279229701970e-02 true resid norm 1.279229701970e-02 ||r(i)||/||b|| 5.190975790704e-03 6 KSP unpreconditioned resid norm 8.642476440830e-03 true resid norm 8.642476440830e-03 ||r(i)||/||b|| 3.507023477252e-03 7 KSP unpreconditioned resid norm 6.257653902253e-03 true resid norm 6.257653902254e-03 ||r(i)||/||b|| 2.539288281313e-03 8 KSP unpreconditioned resid norm 4.265808118282e-03 true resid norm 4.265808118282e-03 ||r(i)||/||b|| 1.731018802619e-03 9 KSP unpreconditioned resid norm 3.197638574989e-03 true resid norm 3.197638574989e-03 ||r(i)||/||b|| 1.297567153469e-03 10 KSP unpreconditioned resid norm 2.541200054047e-03 true resid norm 2.541200054046e-03 ||r(i)||/||b|| 1.031191500602e-03 11 KSP unpreconditioned resid norm 2.076630510557e-03 true resid norm 2.076630510557e-03 ||r(i)||/||b|| 8.426742038540e-04 12 KSP unpreconditioned resid norm 1.800573752044e-03 true resid norm 1.800573752044e-03 ||r(i)||/||b|| 7.306533566131e-04 13 KSP unpreconditioned resid norm 1.525854000374e-03 true resid norm 1.525854000374e-03 ||r(i)||/||b|| 6.191750522905e-04 14 KSP unpreconditioned resid norm 1.295503220832e-03 true resid norm 1.295503220832e-03 ||r(i)||/||b|| 5.257011970377e-04 15 KSP unpreconditioned resid norm 1.084042233066e-03 true resid norm 1.084042233065e-03 ||r(i)||/||b|| 4.398926150069e-04 16 KSP unpreconditioned resid norm 9.072748281531e-04 true resid norm 9.072748281522e-04 ||r(i)||/||b|| 3.681623137111e-04 17 KSP unpreconditioned resid norm 7.055698087217e-04 true resid norm 7.055698087231e-04 ||r(i)||/||b|| 2.863125981278e-04 18 KSP unpreconditioned resid norm 5.647143297836e-04 true resid norm 5.647143297835e-04 ||r(i)||/||b|| 2.291549680292e-04 19 KSP unpreconditioned resid norm 4.500753015708e-04 true resid norm 4.500753015704e-04 ||r(i)||/||b|| 1.826356901225e-04 20 KSP unpreconditioned resid norm 3.751165957785e-04 true resid norm 3.751165957780e-04 ||r(i)||/||b|| 1.522182579388e-04 21 KSP unpreconditioned resid norm 3.100897252663e-04 true resid norm 3.100897252655e-04 ||r(i)||/||b|| 1.258310571057e-04 22 KSP unpreconditioned resid norm 2.538816813673e-04 true resid norm 2.538816813667e-04 ||r(i)||/||b|| 1.030224407429e-04 23 KSP unpreconditioned resid norm 2.056390116417e-04 true resid norm 2.056390116429e-04 ||r(i)||/||b|| 8.344608708027e-05 24 KSP unpreconditioned resid norm 1.608371013495e-04 true resid norm 1.608371013504e-04 ||r(i)||/||b|| 6.526595638539e-05 25 KSP unpreconditioned resid norm 1.253589134032e-04 true resid norm 1.253589134030e-04 ||r(i)||/||b|| 5.086929138854e-05 26 KSP unpreconditioned resid norm 9.775671156099e-05 true resid norm 9.775671156060e-05 ||r(i)||/||b|| 3.966861637971e-05 27 KSP unpreconditioned resid norm 7.341573028295e-05 true resid norm 7.341573028263e-05 ||r(i)||/||b|| 2.979130940808e-05 28 KSP unpreconditioned resid norm 5.564380420345e-05 true resid norm 5.564380420310e-05 ||r(i)||/||b|| 2.257965399616e-05 29 KSP unpreconditioned resid norm 4.323591120259e-05 true resid norm 4.323591120220e-05 ||r(i)||/||b|| 1.754466519922e-05 30 KSP unpreconditioned resid norm 3.368099812126e-05 true resid norm 3.368099812126e-05 ||r(i)||/||b|| 1.366738480078e-05 31 KSP unpreconditioned resid norm 2.990017310739e-05 true resid norm 2.990017310738e-05 ||r(i)||/||b|| 1.213316689717e-05 32 KSP unpreconditioned resid norm 2.518721971644e-05 true resid norm 2.518721971645e-05 ||r(i)||/||b|| 1.022070137848e-05 33 KSP unpreconditioned resid norm 2.135321757357e-05 true resid norm 2.135321757358e-05 ||r(i)||/||b|| 8.664904771001e-06 34 KSP unpreconditioned resid norm 1.708173255141e-05 true resid norm 1.708173255150e-05 ||r(i)||/||b|| 6.931582342212e-06 35 KSP unpreconditioned resid norm 1.378692211645e-05 true resid norm 1.378692211657e-05 ||r(i)||/||b|| 5.594583898825e-06 36 KSP unpreconditioned resid norm 1.075085218635e-05 true resid norm 1.075085218651e-05 ||r(i)||/||b|| 4.362579554216e-06 37 KSP unpreconditioned resid norm 8.579543904874e-06 true resid norm 8.579543905050e-06 ||r(i)||/||b|| 3.481486134805e-06 38 KSP unpreconditioned resid norm 6.663024149842e-06 true resid norm 6.663024149966e-06 ||r(i)||/||b|| 2.703783144034e-06 39 KSP unpreconditioned resid norm 5.488872221286e-06 true resid norm 5.488872221412e-06 ||r(i)||/||b|| 2.227324989072e-06 40 KSP unpreconditioned resid norm 4.402005575508e-06 true resid norm 4.402005575670e-06 ||r(i)||/||b|| 1.786286257945e-06 41 KSP unpreconditioned resid norm 3.615398721527e-06 true resid norm 3.615398721731e-06 ||r(i)||/||b|| 1.467089703229e-06 42 KSP unpreconditioned resid norm 2.967489745085e-06 true resid norm 2.967489745333e-06 ||r(i)||/||b|| 1.204175247296e-06 43 KSP unpreconditioned resid norm 2.471957747140e-06 true resid norm 2.471957747409e-06 ||r(i)||/||b|| 1.003093721376e-06 44 KSP unpreconditioned resid norm 2.088449399093e-06 true resid norm 2.088449399350e-06 ||r(i)||/||b|| 8.474701811129e-07 45 KSP unpreconditioned resid norm 1.755508391147e-06 true resid norm 1.755508391304e-06 ||r(i)||/||b|| 7.123663205757e-07 46 KSP unpreconditioned resid norm 1.489882203160e-06 true resid norm 1.489882203306e-06 ||r(i)||/||b|| 6.045780860507e-07 47 KSP unpreconditioned resid norm 1.213783513857e-06 true resid norm 1.213783513883e-06 ||r(i)||/||b|| 4.925402237003e-07 48 KSP unpreconditioned resid norm 1.021510479234e-06 true resid norm 1.021510479145e-06 ||r(i)||/||b|| 4.145179055040e-07 49 KSP unpreconditioned resid norm 8.287946453344e-07 true resid norm 8.287946452356e-07 ||r(i)||/||b|| 3.363159041927e-07 50 KSP unpreconditioned resid norm 6.668156473414e-07 true resid norm 6.668156471973e-07 ||r(i)||/||b|| 2.705865784802e-07 51 KSP unpreconditioned resid norm 5.280582271016e-07 true resid norm 5.280582269962e-07 ||r(i)||/||b|| 2.142803179286e-07 52 KSP unpreconditioned resid norm 4.157123103826e-07 true resid norm 4.157123102940e-07 ||r(i)||/||b|| 1.686915598747e-07 53 KSP unpreconditioned resid norm 3.368064587348e-07 true resid norm 3.368064586986e-07 ||r(i)||/||b|| 1.366724186098e-07 54 KSP unpreconditioned resid norm 2.710364540227e-07 true resid norm 2.710364539492e-07 ||r(i)||/||b|| 1.099836619398e-07 55 KSP unpreconditioned resid norm 2.066135720810e-07 true resid norm 2.066135719884e-07 ||r(i)||/||b|| 8.384155312928e-08 Linear solve converged due to CONVERGED_RTOL iterations 55 Residual norm 2.06614e-07 Rank#24, time step = 0, continuity = 4.174955e-04 @ (41, 4, 4) Rank#27, time step = 0, continuity = 4.094066e+00 @ (4, 11, 17) Rank#10, time step = 0, continuity = 2.590009e-03 @ (39, 4, 35) Rank#20, time step = 0, continuity = 1.437608e-02 @ (4, 51, 20) Rank#11, time step = 0, continuity = 5.650684e-03 @ (4, 4, 35) Rank#21, time step = 0, continuity = 9.027686e-03 @ (4, 51, 20) Rank#1, time step = 0, continuity = 2.525042e-04 @ (4, 4, 4) Rank#16, time step = 0, continuity = 3.678162e-04 @ (41, 51, 4) Rank#4, time step = 0, continuity = 1.159767e-02 @ (4, 51, 35) Rank#8, time step = 0, continuity = 4.915367e-05 @ (41, 4, 35) Rank#34, time step = 0, continuity = 1.181571e-01 @ (40, 51, 4) Rank#0, time step = 0, continuity = 3.137227e-08 @ (41, 51, 4) Rank#17, time step = 0, continuity = 5.089529e-03 @ (41, 51, 4) Rank#5, time step = 0, continuity = 8.826453e-03 @ (4, 51, 35) Rank#2, time step = 0, continuity = 1.634919e-03 @ (4, 51, 35) Rank#6, time step = 0, continuity = 8.433646e-03 @ (4, 51, 35) Rank#12, time step = 0, continuity = 1.165931e-02 @ (4, 4, 35) Rank#13, time step = 0, continuity = 8.826511e-03 @ (4, 5, 35) Rank#14, time step = 0, continuity = 8.434087e-03 @ (4, 6, 35) Rank#44, time step = 0, continuity = 1.167705e-02 @ (4, 4, 5) Rank#45, time step = 0, continuity = 8.865762e-03 @ (4, 5, 4) Rank#46, time step = 0, continuity = 8.451129e-03 @ (4, 6, 4) Rank#25, time step = 0, continuity = 5.328321e-03 @ (41, 4, 4) Rank#28, time step = 0, continuity = 1.454749e-02 @ (4, 4, 20) Rank#33, time step = 0, continuity = 4.844833e-03 @ (41, 51, 4) Rank#26, time step = 0, continuity = 1.016585e+00 @ (39, 4, 17) Rank#40, time step = 0, continuity = 2.881710e-04 @ (41, 5, 4) Rank#29, time step = 0, continuity = 9.027575e-03 @ (4, 5, 20) Rank#32, time step = 0, continuity = 2.879931e-04 @ (41, 51, 4) Rank#36, time step = 0, continuity = 1.161600e-02 @ (4, 51, 5) Rank#41, time step = 0, continuity = 4.847584e-03 @ (41, 5, 4) Rank#30, time step = 0, continuity = 8.459037e-03 @ (4, 6, 22) Rank#37, time step = 0, continuity = 8.865921e-03 @ (4, 51, 4) Rank#42, time step = 0, continuity = 1.182950e-01 @ (40, 5, 4) Rank#31, time step = 0, continuity = 8.378688e-03 @ (4, 12, 28) Rank#38, time step = 0, continuity = 8.450654e-03 @ (4, 51, 4) Rank#43, time step = 0, continuity = 1.077929e-01 @ (4, 5, 4) Rank#39, time step = 0, continuity = 8.377602e-03 @ (4, 51, 5) Rank#35, time step = 0, continuity = 1.076056e-01 @ (4, 51, 4) Rank#18, time step = 0, continuity = 9.940864e-02 @ (40, 51, 4) Rank#9, time step = 0, continuity = 2.400260e-04 @ (4, 4, 35) Rank#23, time step = 0, continuity = 8.378040e-03 @ (4, 51, 29) Rank#22, time step = 0, continuity = 8.458582e-03 @ (4, 51, 22) Rank#47, time step = 0, continuity = 8.378259e-03 @ (4, 12, 5) Rank#3, time step = 0, continuity = 3.642321e-03 @ (4, 4, 4) Rank#7, time step = 0, continuity = 8.373002e-03 @ (4, 51, 35) Rank#15, time step = 0, continuity = 8.373655e-03 @ (4, 12, 35) Rank#19, time step = 0, continuity = 4.094089e+00 @ (4, 45, 17) Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 Current time step = 1, Total time step number = 5 0 KSP unpreconditioned resid norm 7.406650929703e+00 true resid norm 7.406650929703e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 5.561334348759e-01 true resid norm 5.561334348760e-01 ||r(i)||/||b|| 7.508568179523e-02 2 KSP unpreconditioned resid norm 1.554698924650e-01 true resid norm 1.554698924650e-01 ||r(i)||/||b|| 2.099057913497e-02 3 KSP unpreconditioned resid norm 5.829927905115e-02 true resid norm 5.829927905115e-02 ||r(i)||/||b|| 7.871206514857e-03 4 KSP unpreconditioned resid norm 3.294371415430e-02 true resid norm 3.294371415430e-02 ||r(i)||/||b|| 4.447855645821e-03 5 KSP unpreconditioned resid norm 2.126154585895e-02 true resid norm 2.126154585896e-02 ||r(i)||/||b|| 2.870601849709e-03 6 KSP unpreconditioned resid norm 1.282632551143e-02 true resid norm 1.282632551143e-02 ||r(i)||/||b|| 1.731730796168e-03 7 KSP unpreconditioned resid norm 8.505640366786e-03 true resid norm 8.505640366788e-03 ||r(i)||/||b|| 1.148378727108e-03 8 KSP unpreconditioned resid norm 6.213706906443e-03 true resid norm 6.213706906443e-03 ||r(i)||/||b|| 8.389361082921e-04 9 KSP unpreconditioned resid norm 4.308546423382e-03 true resid norm 4.308546423384e-03 ||r(i)||/||b|| 5.817131743181e-04 10 KSP unpreconditioned resid norm 3.252806301386e-03 true resid norm 3.252806301386e-03 ||r(i)||/||b|| 4.391737010774e-04 11 KSP unpreconditioned resid norm 2.550371639217e-03 true resid norm 2.550371639217e-03 ||r(i)||/||b|| 3.443353363649e-04 12 KSP unpreconditioned resid norm 1.973664425127e-03 true resid norm 1.973664425127e-03 ||r(i)||/||b|| 2.664719107002e-04 13 KSP unpreconditioned resid norm 1.548366484711e-03 true resid norm 1.548366484710e-03 ||r(i)||/||b|| 2.090508246448e-04 14 KSP unpreconditioned resid norm 1.253035127943e-03 true resid norm 1.253035127943e-03 ||r(i)||/||b|| 1.691770193891e-04 15 KSP unpreconditioned resid norm 9.627823525618e-04 true resid norm 9.627823525611e-04 ||r(i)||/||b|| 1.299888926451e-04 16 KSP unpreconditioned resid norm 7.830570268883e-04 true resid norm 7.830570268888e-04 ||r(i)||/||b|| 1.057234955881e-04 17 KSP unpreconditioned resid norm 6.258324642484e-04 true resid norm 6.258324642482e-04 ||r(i)||/||b|| 8.449601178563e-05 18 KSP unpreconditioned resid norm 5.131714996563e-04 true resid norm 5.131714996560e-04 ||r(i)||/||b|| 6.928522817216e-05 19 KSP unpreconditioned resid norm 4.157415220129e-04 true resid norm 4.157415220130e-04 ||r(i)||/||b|| 5.613083780495e-05 20 KSP unpreconditioned resid norm 3.503809296735e-04 true resid norm 3.503809296738e-04 ||r(i)||/||b|| 4.730625663330e-05 21 KSP unpreconditioned resid norm 2.855345717794e-04 true resid norm 2.855345717803e-04 ||r(i)||/||b|| 3.855110420220e-05 22 KSP unpreconditioned resid norm 2.384533498881e-04 true resid norm 2.384533498885e-04 ||r(i)||/||b|| 3.219449008084e-05 23 KSP unpreconditioned resid norm 1.934365375858e-04 true resid norm 1.934365375864e-04 ||r(i)||/||b|| 2.611659971859e-05 24 KSP unpreconditioned resid norm 1.510009889378e-04 true resid norm 1.510009889395e-04 ||r(i)||/||b|| 2.038721554083e-05 25 KSP unpreconditioned resid norm 1.178370506710e-04 true resid norm 1.178370506724e-04 ||r(i)||/||b|| 1.590962660328e-05 26 KSP unpreconditioned resid norm 8.921055829265e-05 true resid norm 8.921055829443e-05 ||r(i)||/||b|| 1.204465542404e-05 27 KSP unpreconditioned resid norm 6.903213040845e-05 true resid norm 6.903213041012e-05 ||r(i)||/||b|| 9.320289435172e-06 28 KSP unpreconditioned resid norm 5.357460942945e-05 true resid norm 5.357460943198e-05 ||r(i)||/||b|| 7.233310971512e-06 29 KSP unpreconditioned resid norm 4.182989256886e-05 true resid norm 4.182989257160e-05 ||r(i)||/||b|| 5.647612256688e-06 30 KSP unpreconditioned resid norm 3.087441942016e-05 true resid norm 3.087441942016e-05 ||r(i)||/||b|| 4.168472324833e-06 31 KSP unpreconditioned resid norm 2.643983394241e-05 true resid norm 2.643983394242e-05 ||r(i)||/||b|| 3.569742140323e-06 32 KSP unpreconditioned resid norm 2.216484802712e-05 true resid norm 2.216484802713e-05 ||r(i)||/||b|| 2.992560097336e-06 33 KSP unpreconditioned resid norm 1.907523007213e-05 true resid norm 1.907523007214e-05 ||r(i)||/||b|| 2.575419073098e-06 34 KSP unpreconditioned resid norm 1.547185278986e-05 true resid norm 1.547185278987e-05 ||r(i)||/||b|| 2.088913455854e-06 35 KSP unpreconditioned resid norm 1.256379017347e-05 true resid norm 1.256379017349e-05 ||r(i)||/||b|| 1.696284905652e-06 36 KSP unpreconditioned resid norm 1.072584068780e-05 true resid norm 1.072584068780e-05 ||r(i)||/||b|| 1.448136383042e-06 37 KSP unpreconditioned resid norm 9.211701529799e-06 true resid norm 9.211701529800e-06 ||r(i)||/||b|| 1.243706719438e-06 38 KSP unpreconditioned resid norm 8.028399995262e-06 true resid norm 8.028399995246e-06 ||r(i)||/||b|| 1.083944696658e-06 39 KSP unpreconditioned resid norm 6.857325667061e-06 true resid norm 6.857325667056e-06 ||r(i)||/||b|| 9.258335153282e-07 40 KSP unpreconditioned resid norm 5.759722349850e-06 true resid norm 5.759722349846e-06 ||r(i)||/||b|| 7.776419335152e-07 41 KSP unpreconditioned resid norm 4.722558786184e-06 true resid norm 4.722558786179e-06 ||r(i)||/||b|| 6.376105517867e-07 42 KSP unpreconditioned resid norm 3.797562376720e-06 true resid norm 3.797562376716e-06 ||r(i)||/||b|| 5.127232824604e-07 43 KSP unpreconditioned resid norm 2.942658206923e-06 true resid norm 2.942658206913e-06 ||r(i)||/||b|| 3.972994319352e-07 44 KSP unpreconditioned resid norm 2.251149576887e-06 true resid norm 2.251149576873e-06 ||r(i)||/||b|| 3.039362322107e-07 45 KSP unpreconditioned resid norm 1.698940938700e-06 true resid norm 1.698940938710e-06 ||r(i)||/||b|| 2.293804520876e-07 46 KSP unpreconditioned resid norm 1.261300739337e-06 true resid norm 1.261300739358e-06 ||r(i)||/||b|| 1.702929908983e-07 47 KSP unpreconditioned resid norm 9.471923629982e-07 true resid norm 9.471923630326e-07 ||r(i)||/||b|| 1.278840291007e-07 48 KSP unpreconditioned resid norm 7.128407378936e-07 true resid norm 7.128407379083e-07 ||r(i)||/||b|| 9.624332841847e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 7.12841e-07 0 KSP unpreconditioned resid norm 2.178065518847e+00 true resid norm 2.178065518847e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.460392868474e-01 true resid norm 1.460392868474e-01 ||r(i)||/||b|| 6.704999715746e-02 2 KSP unpreconditioned resid norm 3.941062007261e-02 true resid norm 3.941062007261e-02 ||r(i)||/||b|| 1.809432256817e-02 3 KSP unpreconditioned resid norm 1.469001086082e-02 true resid norm 1.469001086082e-02 ||r(i)||/||b|| 6.744522023654e-03 4 KSP unpreconditioned resid norm 7.045501244561e-03 true resid norm 7.045501244561e-03 ||r(i)||/||b|| 3.234751748098e-03 5 KSP unpreconditioned resid norm 4.607801857748e-03 true resid norm 4.607801857748e-03 ||r(i)||/||b|| 2.115547864781e-03 6 KSP unpreconditioned resid norm 2.920588014185e-03 true resid norm 2.920588014185e-03 ||r(i)||/||b|| 1.340909164078e-03 7 KSP unpreconditioned resid norm 1.837118592260e-03 true resid norm 1.837118592261e-03 ||r(i)||/||b|| 8.434634203445e-04 8 KSP unpreconditioned resid norm 1.326010801107e-03 true resid norm 1.326010801107e-03 ||r(i)||/||b|| 6.088020721291e-04 9 KSP unpreconditioned resid norm 9.748366701558e-04 true resid norm 9.748366701566e-04 ||r(i)||/||b|| 4.475699476079e-04 10 KSP unpreconditioned resid norm 7.331750895867e-04 true resid norm 7.331750895872e-04 ||r(i)||/||b|| 3.366175550015e-04 11 KSP unpreconditioned resid norm 5.814568566459e-04 true resid norm 5.814568566461e-04 ||r(i)||/||b|| 2.669602230120e-04 12 KSP unpreconditioned resid norm 5.038353768751e-04 true resid norm 5.038353768752e-04 ||r(i)||/||b|| 2.313224154717e-04 13 KSP unpreconditioned resid norm 4.261654180571e-04 true resid norm 4.261654180568e-04 ||r(i)||/||b|| 1.956623500851e-04 14 KSP unpreconditioned resid norm 3.777823804727e-04 true resid norm 3.777823804721e-04 ||r(i)||/||b|| 1.734485841694e-04 15 KSP unpreconditioned resid norm 3.149422832642e-04 true resid norm 3.149422832633e-04 ||r(i)||/||b|| 1.445972495034e-04 16 KSP unpreconditioned resid norm 2.738474743364e-04 true resid norm 2.738474743356e-04 ||r(i)||/||b|| 1.257296770763e-04 17 KSP unpreconditioned resid norm 2.130851183760e-04 true resid norm 2.130851183749e-04 ||r(i)||/||b|| 9.783228122895e-05 18 KSP unpreconditioned resid norm 1.744002705546e-04 true resid norm 1.744002705535e-04 ||r(i)||/||b|| 8.007117740235e-05 19 KSP unpreconditioned resid norm 1.268436039537e-04 true resid norm 1.268436039527e-04 ||r(i)||/||b|| 5.823681742132e-05 20 KSP unpreconditioned resid norm 1.025539525209e-04 true resid norm 1.025539525202e-04 ||r(i)||/||b|| 4.708487951018e-05 21 KSP unpreconditioned resid norm 8.111346657473e-05 true resid norm 8.111346657392e-05 ||r(i)||/||b|| 3.724105903705e-05 22 KSP unpreconditioned resid norm 6.811645792110e-05 true resid norm 6.811645792094e-05 ||r(i)||/||b|| 3.127383328533e-05 23 KSP unpreconditioned resid norm 5.476915383296e-05 true resid norm 5.476915383309e-05 ||r(i)||/||b|| 2.514577883869e-05 24 KSP unpreconditioned resid norm 4.361783613715e-05 true resid norm 4.361783613789e-05 ||r(i)||/||b|| 2.002595227759e-05 25 KSP unpreconditioned resid norm 3.473686153395e-05 true resid norm 3.473686153498e-05 ||r(i)||/||b|| 1.594849247390e-05 26 KSP unpreconditioned resid norm 2.662103935785e-05 true resid norm 2.662103936019e-05 ||r(i)||/||b|| 1.222233175717e-05 27 KSP unpreconditioned resid norm 2.200010814357e-05 true resid norm 2.200010814643e-05 ||r(i)||/||b|| 1.010075590292e-05 28 KSP unpreconditioned resid norm 1.774330318901e-05 true resid norm 1.774330319304e-05 ||r(i)||/||b|| 8.146358793851e-06 29 KSP unpreconditioned resid norm 1.494333750491e-05 true resid norm 1.494333750969e-05 ||r(i)||/||b|| 6.860830117544e-06 30 KSP unpreconditioned resid norm 1.206656267301e-05 true resid norm 1.206656267301e-05 ||r(i)||/||b|| 5.540036591461e-06 31 KSP unpreconditioned resid norm 1.120278995565e-05 true resid norm 1.120278995564e-05 ||r(i)||/||b|| 5.143458660314e-06 32 KSP unpreconditioned resid norm 9.261235878008e-06 true resid norm 9.261235878003e-06 ||r(i)||/||b|| 4.252046505427e-06 33 KSP unpreconditioned resid norm 8.197470936141e-06 true resid norm 8.197470936131e-06 ||r(i)||/||b|| 3.763647541913e-06 34 KSP unpreconditioned resid norm 6.820318077282e-06 true resid norm 6.820318077276e-06 ||r(i)||/||b|| 3.131364974221e-06 35 KSP unpreconditioned resid norm 5.688691374400e-06 true resid norm 5.688691374394e-06 ||r(i)||/||b|| 2.611809114634e-06 36 KSP unpreconditioned resid norm 4.595604182694e-06 true resid norm 4.595604182687e-06 ||r(i)||/||b|| 2.109947631475e-06 37 KSP unpreconditioned resid norm 3.771797432721e-06 true resid norm 3.771797432722e-06 ||r(i)||/||b|| 1.731718995634e-06 38 KSP unpreconditioned resid norm 3.109066412989e-06 true resid norm 3.109066412982e-06 ||r(i)||/||b|| 1.427443934115e-06 39 KSP unpreconditioned resid norm 2.669628603745e-06 true resid norm 2.669628603739e-06 ||r(i)||/||b|| 1.225687923820e-06 40 KSP unpreconditioned resid norm 2.210365942259e-06 true resid norm 2.210365942256e-06 ||r(i)||/||b|| 1.014829867664e-06 41 KSP unpreconditioned resid norm 1.821949773643e-06 true resid norm 1.821949773631e-06 ||r(i)||/||b|| 8.364990666563e-07 42 KSP unpreconditioned resid norm 1.446964504975e-06 true resid norm 1.446964504973e-06 ||r(i)||/||b|| 6.643347008862e-07 43 KSP unpreconditioned resid norm 1.161968636199e-06 true resid norm 1.161968636218e-06 ||r(i)||/||b|| 5.334865393916e-07 44 KSP unpreconditioned resid norm 9.487319749258e-07 true resid norm 9.487319749359e-07 ||r(i)||/||b|| 4.355846813269e-07 45 KSP unpreconditioned resid norm 7.697968173503e-07 true resid norm 7.697968173499e-07 ||r(i)||/||b|| 3.534314329339e-07 46 KSP unpreconditioned resid norm 6.509475897671e-07 true resid norm 6.509475897554e-07 ||r(i)||/||b|| 2.988650176603e-07 47 KSP unpreconditioned resid norm 5.418206977645e-07 true resid norm 5.418206977467e-07 ||r(i)||/||b|| 2.487623503785e-07 48 KSP unpreconditioned resid norm 4.629579528470e-07 true resid norm 4.629579528207e-07 ||r(i)||/||b|| 2.125546494422e-07 49 KSP unpreconditioned resid norm 3.742046372363e-07 true resid norm 3.742046372009e-07 ||r(i)||/||b|| 1.718059599048e-07 50 KSP unpreconditioned resid norm 2.906109875748e-07 true resid norm 2.906109875442e-07 ||r(i)||/||b|| 1.334261917420e-07 51 KSP unpreconditioned resid norm 2.167356702683e-07 true resid norm 2.167356702358e-07 ||r(i)||/||b|| 9.950833359255e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 2.16736e-07 Rank#2, time step = 1, continuity = 2.025209e-03 @ (4, 51, 35) Rank#44, time step = 1, continuity = 8.147067e-03 @ (5, 5, 4) Rank#24, time step = 1, continuity = 3.314627e-04 @ (41, 4, 4) Rank#10, time step = 1, continuity = 2.015249e-03 @ (4, 5, 35) Rank#1, time step = 1, continuity = 9.967610e-05 @ (4, 4, 4) Rank#46, time step = 1, continuity = 2.260864e-03 @ (4, 50, 34) Rank#27, time step = 1, continuity = 4.478382e+00 @ (4, 11, 17) Rank#11, time step = 1, continuity = 3.502620e-02 @ (4, 4, 35) Rank#31, time step = 1, continuity = 2.189822e-03 @ (4, 50, 5) Rank#0, time step = 1, continuity = 3.535561e-08 @ (19, 51, 7) Rank#47, time step = 1, continuity = 2.189827e-03 @ (4, 50, 34) Rank#8, time step = 1, continuity = 3.448571e-05 @ (41, 4, 35) Rank#30, time step = 1, continuity = 2.226857e-03 @ (4, 50, 5) Rank#4, time step = 1, continuity = 4.905157e-03 @ (4, 4, 4) Rank#28, time step = 1, continuity = 1.385920e-02 @ (4, 4, 20) Rank#5, time step = 1, continuity = 2.806331e-03 @ (4, 4, 4) Rank#25, time step = 1, continuity = 4.553406e-03 @ (41, 4, 4) Rank#29, time step = 1, continuity = 2.459065e-03 @ (4, 50, 5) Rank#3, time step = 1, continuity = 3.336243e-02 @ (4, 51, 35) Rank#6, time step = 1, continuity = 2.297121e-03 @ (4, 4, 4) Rank#26, time step = 1, continuity = 9.589653e-01 @ (39, 4, 17) Rank#20, time step = 1, continuity = 1.330094e-02 @ (4, 51, 20) Rank#21, time step = 1, continuity = 2.511545e-03 @ (4, 4, 5) Rank#16, time step = 1, continuity = 2.985854e-04 @ (41, 51, 4) Rank#37, time step = 1, continuity = 2.769053e-03 @ (4, 4, 34) Rank#40, time step = 1, continuity = 2.449979e-04 @ (41, 5, 4) Rank#12, time step = 1, continuity = 4.912665e-03 @ (4, 50, 4) Rank#22, time step = 1, continuity = 2.246064e-03 @ (4, 4, 5) Rank#17, time step = 1, continuity = 4.362478e-03 @ (41, 51, 4) Rank#36, time step = 1, continuity = 8.123727e-03 @ (5, 51, 4) Rank#41, time step = 1, continuity = 4.216866e-03 @ (41, 5, 4) Rank#14, time step = 1, continuity = 2.279742e-03 @ (4, 50, 4) Rank#23, time step = 1, continuity = 2.197173e-03 @ (4, 4, 5) Rank#18, time step = 1, continuity = 8.029293e-02 @ (39, 51, 4) Rank#38, time step = 1, continuity = 2.278785e-03 @ (4, 4, 34) Rank#42, time step = 1, continuity = 9.597068e-02 @ (40, 5, 4) Rank#15, time step = 1, continuity = 2.197458e-03 @ (4, 50, 4) Rank#19, time step = 1, continuity = 4.478201e+00 @ (4, 45, 17) Rank#43, time step = 1, continuity = 8.285460e-02 @ (19, 5, 4) Rank#39, time step = 1, continuity = 2.197159e-03 @ (4, 4, 34) Rank#45, time step = 1, continuity = 2.732214e-03 @ (4, 50, 34) Rank#32, time step = 1, continuity = 2.455089e-04 @ (41, 51, 4) Rank#33, time step = 1, continuity = 4.212290e-03 @ (41, 51, 4) Rank#34, time step = 1, continuity = 9.554038e-02 @ (40, 51, 4) Rank#9, time step = 1, continuity = 1.731917e-04 @ (41, 4, 35) Rank#13, time step = 1, continuity = 2.771882e-03 @ (4, 50, 4) Rank#7, time step = 1, continuity = 2.204604e-03 @ (4, 4, 4) Rank#35, time step = 1, continuity = 8.228460e-02 @ (19, 51, 4) Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 Current time step = 2, Total time step number = 5 0 KSP unpreconditioned resid norm 1.145993263262e+01 true resid norm 1.145993263262e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 8.467018295731e-01 true resid norm 8.467018295732e-01 ||r(i)||/||b|| 7.388366552551e-02 2 KSP unpreconditioned resid norm 2.353082733202e-01 true resid norm 2.353082733202e-01 ||r(i)||/||b|| 2.053312884671e-02 3 KSP unpreconditioned resid norm 8.280650623931e-02 true resid norm 8.280650623932e-02 ||r(i)||/||b|| 7.225741101098e-03 4 KSP unpreconditioned resid norm 4.146387000857e-02 true resid norm 4.146387000857e-02 ||r(i)||/||b|| 3.618160013485e-03 5 KSP unpreconditioned resid norm 2.760201858697e-02 true resid norm 2.760201858697e-02 ||r(i)||/||b|| 2.408567264035e-03 6 KSP unpreconditioned resid norm 1.750509619293e-02 true resid norm 1.750509619293e-02 ||r(i)||/||b|| 1.527504284197e-03 7 KSP unpreconditioned resid norm 1.101437000383e-02 true resid norm 1.101437000383e-02 ||r(i)||/||b|| 9.611199609047e-04 8 KSP unpreconditioned resid norm 8.020603798473e-03 true resid norm 8.020603798472e-03 ||r(i)||/||b|| 6.998822816501e-04 9 KSP unpreconditioned resid norm 5.769911324715e-03 true resid norm 5.769911324718e-03 ||r(i)||/||b|| 5.034856233182e-04 10 KSP unpreconditioned resid norm 4.217969701212e-03 true resid norm 4.217969701212e-03 ||r(i)||/||b|| 3.680623469991e-04 11 KSP unpreconditioned resid norm 3.290098525645e-03 true resid norm 3.290098525644e-03 ||r(i)||/||b|| 2.870957998723e-04 12 KSP unpreconditioned resid norm 2.609406224541e-03 true resid norm 2.609406224541e-03 ||r(i)||/||b|| 2.276982167517e-04 13 KSP unpreconditioned resid norm 2.004708786049e-03 true resid norm 2.004708786048e-03 ||r(i)||/||b|| 1.749319869772e-04 14 KSP unpreconditioned resid norm 1.621432183334e-03 true resid norm 1.621432183333e-03 ||r(i)||/||b|| 1.414870606410e-04 15 KSP unpreconditioned resid norm 1.273824611796e-03 true resid norm 1.273824611795e-03 ||r(i)||/||b|| 1.111546335071e-04 16 KSP unpreconditioned resid norm 1.022867010282e-03 true resid norm 1.022867010282e-03 ||r(i)||/||b|| 8.925593570862e-05 17 KSP unpreconditioned resid norm 8.088159574836e-04 true resid norm 8.088159574832e-04 ||r(i)||/||b|| 7.057772357066e-05 18 KSP unpreconditioned resid norm 6.782535622143e-04 true resid norm 6.782535622147e-04 ||r(i)||/||b|| 5.918477742916e-05 19 KSP unpreconditioned resid norm 5.390023303178e-04 true resid norm 5.390023303189e-04 ||r(i)||/||b|| 4.703363864328e-05 20 KSP unpreconditioned resid norm 4.544503585906e-04 true resid norm 4.544503585916e-04 ||r(i)||/||b|| 3.965558726743e-05 21 KSP unpreconditioned resid norm 3.748239986292e-04 true resid norm 3.748239986313e-04 ||r(i)||/||b|| 3.270734747292e-05 22 KSP unpreconditioned resid norm 3.106950604455e-04 true resid norm 3.106950604470e-04 ||r(i)||/||b|| 2.711142119306e-05 23 KSP unpreconditioned resid norm 2.519104118885e-04 true resid norm 2.519104118907e-04 ||r(i)||/||b|| 2.198184055408e-05 24 KSP unpreconditioned resid norm 1.993796457577e-04 true resid norm 1.993796457610e-04 ||r(i)||/||b|| 1.739797712192e-05 25 KSP unpreconditioned resid norm 1.536074482824e-04 true resid norm 1.536074482846e-04 ||r(i)||/||b|| 1.340387009321e-05 26 KSP unpreconditioned resid norm 1.160166482479e-04 true resid norm 1.160166482505e-04 ||r(i)||/||b|| 1.012367628762e-05 27 KSP unpreconditioned resid norm 9.118828433660e-05 true resid norm 9.118828433948e-05 ||r(i)||/||b|| 7.957139650188e-06 28 KSP unpreconditioned resid norm 6.968237176481e-05 true resid norm 6.968237176823e-05 ||r(i)||/||b|| 6.080521936918e-06 29 KSP unpreconditioned resid norm 5.489756729503e-05 true resid norm 5.489756729856e-05 ||r(i)||/||b|| 4.790391798841e-06 30 KSP unpreconditioned resid norm 4.088864972904e-05 true resid norm 4.088864972904e-05 ||r(i)||/||b|| 3.567965976751e-06 31 KSP unpreconditioned resid norm 3.550536673258e-05 true resid norm 3.550536673258e-05 ||r(i)||/||b|| 3.098217753177e-06 32 KSP unpreconditioned resid norm 2.932517724995e-05 true resid norm 2.932517724995e-05 ||r(i)||/||b|| 2.558931033023e-06 33 KSP unpreconditioned resid norm 2.538035496051e-05 true resid norm 2.538035496053e-05 ||r(i)||/||b|| 2.214703678824e-06 34 KSP unpreconditioned resid norm 2.077966563835e-05 true resid norm 2.077966563835e-05 ||r(i)||/||b|| 1.813245008021e-06 35 KSP unpreconditioned resid norm 1.677227950577e-05 true resid norm 1.677227950575e-05 ||r(i)||/||b|| 1.463558298590e-06 36 KSP unpreconditioned resid norm 1.426971881665e-05 true resid norm 1.426971881664e-05 ||r(i)||/||b|| 1.245183481797e-06 37 KSP unpreconditioned resid norm 1.232421452779e-05 true resid norm 1.232421452778e-05 ||r(i)||/||b|| 1.075417711681e-06 38 KSP unpreconditioned resid norm 1.072305282161e-05 true resid norm 1.072305282161e-05 ||r(i)||/||b|| 9.356994639823e-07 39 KSP unpreconditioned resid norm 9.187459115632e-06 true resid norm 9.187459115628e-06 ||r(i)||/||b|| 8.017027159025e-07 40 KSP unpreconditioned resid norm 7.725611104268e-06 true resid norm 7.725611104266e-06 ||r(i)||/||b|| 6.741410575380e-07 41 KSP unpreconditioned resid norm 6.336099247960e-06 true resid norm 6.336099247978e-06 ||r(i)||/||b|| 5.528914916955e-07 42 KSP unpreconditioned resid norm 5.072527467147e-06 true resid norm 5.072527467184e-06 ||r(i)||/||b|| 4.426315258387e-07 43 KSP unpreconditioned resid norm 3.980186570456e-06 true resid norm 3.980186570487e-06 ||r(i)||/||b|| 3.473132607393e-07 44 KSP unpreconditioned resid norm 3.031763132156e-06 true resid norm 3.031763132231e-06 ||r(i)||/||b|| 2.645533119106e-07 45 KSP unpreconditioned resid norm 2.268918039035e-06 true resid norm 2.268918039109e-06 ||r(i)||/||b|| 1.979870311498e-07 46 KSP unpreconditioned resid norm 1.714692473145e-06 true resid norm 1.714692473173e-06 ||r(i)||/||b|| 1.496250046263e-07 47 KSP unpreconditioned resid norm 1.278527011803e-06 true resid norm 1.278527011797e-06 ||r(i)||/||b|| 1.115649675076e-07 48 KSP unpreconditioned resid norm 9.599919523504e-07 true resid norm 9.599919523332e-07 ||r(i)||/||b|| 8.376942370506e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 9.59992e-07 0 KSP unpreconditioned resid norm 1.903872986821e+00 true resid norm 1.903872986821e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.250166965029e-01 true resid norm 1.250166965029e-01 ||r(i)||/||b|| 6.566441005689e-02 2 KSP unpreconditioned resid norm 3.413148600032e-02 true resid norm 3.413148600032e-02 ||r(i)||/||b|| 1.792739654199e-02 3 KSP unpreconditioned resid norm 1.224574003913e-02 true resid norm 1.224574003913e-02 ||r(i)||/||b|| 6.432015225750e-03 4 KSP unpreconditioned resid norm 5.300073743184e-03 true resid norm 5.300073743185e-03 ||r(i)||/||b|| 2.783837881976e-03 5 KSP unpreconditioned resid norm 3.365317369016e-03 true resid norm 3.365317369016e-03 ||r(i)||/||b|| 1.767616533409e-03 6 KSP unpreconditioned resid norm 2.233194123812e-03 true resid norm 2.233194123811e-03 ||r(i)||/||b|| 1.172974320908e-03 7 KSP unpreconditioned resid norm 1.414030484365e-03 true resid norm 1.414030484365e-03 ||r(i)||/||b|| 7.427126148400e-04 8 KSP unpreconditioned resid norm 1.008018647560e-03 true resid norm 1.008018647560e-03 ||r(i)||/||b|| 5.294568779209e-04 9 KSP unpreconditioned resid norm 7.859431037379e-04 true resid norm 7.859431037391e-04 ||r(i)||/||b|| 4.128127817241e-04 10 KSP unpreconditioned resid norm 6.142350651412e-04 true resid norm 6.142350651422e-04 ||r(i)||/||b|| 3.226239719739e-04 11 KSP unpreconditioned resid norm 5.010222708685e-04 true resid norm 5.010222708691e-04 ||r(i)||/||b|| 2.631595039886e-04 12 KSP unpreconditioned resid norm 4.370374667634e-04 true resid norm 4.370374667636e-04 ||r(i)||/||b|| 2.295517977243e-04 13 KSP unpreconditioned resid norm 3.751450978703e-04 true resid norm 3.751450978698e-04 ||r(i)||/||b|| 1.970431328490e-04 14 KSP unpreconditioned resid norm 3.234752378984e-04 true resid norm 3.234752378973e-04 ||r(i)||/||b|| 1.699037909232e-04 15 KSP unpreconditioned resid norm 2.685396778927e-04 true resid norm 2.685396778914e-04 ||r(i)||/||b|| 1.410491559837e-04 16 KSP unpreconditioned resid norm 2.309017226947e-04 true resid norm 2.309017226933e-04 ||r(i)||/||b|| 1.212800035988e-04 17 KSP unpreconditioned resid norm 1.762487331106e-04 true resid norm 1.762487331090e-04 ||r(i)||/||b|| 9.257378739494e-05 18 KSP unpreconditioned resid norm 1.448381331042e-04 true resid norm 1.448381331027e-04 ||r(i)||/||b|| 7.607552294995e-05 19 KSP unpreconditioned resid norm 1.077151882122e-04 true resid norm 1.077151882107e-04 ||r(i)||/||b|| 5.657687721623e-05 20 KSP unpreconditioned resid norm 8.753358844749e-05 true resid norm 8.753358844627e-05 ||r(i)||/||b|| 4.597659037771e-05 21 KSP unpreconditioned resid norm 6.789005163891e-05 true resid norm 6.789005163755e-05 ||r(i)||/||b|| 3.565891848221e-05 22 KSP unpreconditioned resid norm 5.730497317295e-05 true resid norm 5.730497317199e-05 ||r(i)||/||b|| 3.009915764794e-05 23 KSP unpreconditioned resid norm 4.488629127859e-05 true resid norm 4.488629127750e-05 ||r(i)||/||b|| 2.357630555620e-05 24 KSP unpreconditioned resid norm 3.549059931238e-05 true resid norm 3.549059931175e-05 ||r(i)||/||b|| 1.864126417960e-05 25 KSP unpreconditioned resid norm 2.825201934536e-05 true resid norm 2.825201934469e-05 ||r(i)||/||b|| 1.483923535879e-05 26 KSP unpreconditioned resid norm 2.204358958226e-05 true resid norm 2.204358958227e-05 ||r(i)||/||b|| 1.157828790831e-05 27 KSP unpreconditioned resid norm 1.826364573014e-05 true resid norm 1.826364573052e-05 ||r(i)||/||b|| 9.592890837228e-06 28 KSP unpreconditioned resid norm 1.488023103673e-05 true resid norm 1.488023103794e-05 ||r(i)||/||b|| 7.815768772885e-06 29 KSP unpreconditioned resid norm 1.283432364115e-05 true resid norm 1.283432364331e-05 ||r(i)||/||b|| 6.741165892973e-06 30 KSP unpreconditioned resid norm 1.014605282698e-05 true resid norm 1.014605282698e-05 ||r(i)||/||b|| 5.329164758998e-06 31 KSP unpreconditioned resid norm 9.407914973543e-06 true resid norm 9.407914973543e-06 ||r(i)||/||b|| 4.941461451822e-06 32 KSP unpreconditioned resid norm 7.757298584563e-06 true resid norm 7.757298584557e-06 ||r(i)||/||b|| 4.074483244551e-06 33 KSP unpreconditioned resid norm 6.931504610008e-06 true resid norm 6.931504610005e-06 ||r(i)||/||b|| 3.640738987309e-06 34 KSP unpreconditioned resid norm 5.698043511339e-06 true resid norm 5.698043511334e-06 ||r(i)||/||b|| 2.992869561560e-06 35 KSP unpreconditioned resid norm 4.862023768014e-06 true resid norm 4.862023768017e-06 ||r(i)||/||b|| 2.553754269152e-06 36 KSP unpreconditioned resid norm 3.935139896939e-06 true resid norm 3.935139896942e-06 ||r(i)||/||b|| 2.066913036837e-06 37 KSP unpreconditioned resid norm 3.257659841958e-06 true resid norm 3.257659841967e-06 ||r(i)||/||b|| 1.711069942437e-06 38 KSP unpreconditioned resid norm 2.658817764230e-06 true resid norm 2.658817764249e-06 ||r(i)||/||b|| 1.396531062026e-06 39 KSP unpreconditioned resid norm 2.259760363675e-06 true resid norm 2.259760363695e-06 ||r(i)||/||b|| 1.186928108827e-06 40 KSP unpreconditioned resid norm 1.850320661460e-06 true resid norm 1.850320661493e-06 ||r(i)||/||b|| 9.718719023281e-07 41 KSP unpreconditioned resid norm 1.507757223827e-06 true resid norm 1.507757223866e-06 ||r(i)||/||b|| 7.919421276016e-07 42 KSP unpreconditioned resid norm 1.222830298328e-06 true resid norm 1.222830298355e-06 ||r(i)||/||b|| 6.422856497358e-07 43 KSP unpreconditioned resid norm 9.791313605986e-07 true resid norm 9.791313606237e-07 ||r(i)||/||b|| 5.142839713580e-07 44 KSP unpreconditioned resid norm 8.030348922423e-07 true resid norm 8.030348922612e-07 ||r(i)||/||b|| 4.217901602785e-07 45 KSP unpreconditioned resid norm 6.450601933616e-07 true resid norm 6.450601933832e-07 ||r(i)||/||b|| 3.388147202299e-07 46 KSP unpreconditioned resid norm 5.406035117950e-07 true resid norm 5.406035118192e-07 ||r(i)||/||b|| 2.839493577362e-07 47 KSP unpreconditioned resid norm 4.409627148245e-07 true resid norm 4.409627148583e-07 ||r(i)||/||b|| 2.316135151403e-07 48 KSP unpreconditioned resid norm 3.776631107440e-07 true resid norm 3.776631107863e-07 ||r(i)||/||b|| 1.983657068515e-07 49 KSP unpreconditioned resid norm 3.060662042891e-07 true resid norm 3.060662043298e-07 ||r(i)||/||b|| 1.607597809562e-07 50 KSP unpreconditioned resid norm 2.450411530613e-07 true resid norm 2.450411530946e-07 ||r(i)||/||b|| 1.287066704506e-07 51 KSP unpreconditioned resid norm 1.892344498531e-07 true resid norm 1.892344498921e-07 ||r(i)||/||b|| 9.939447179615e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 1.89234e-07 Rank#24, time step = 2, continuity = 3.572595e-04 @ (41, 4, 4) Rank#44, time step = 2, continuity = 1.147952e-02 @ (5, 5, 4) Rank#27, time step = 2, continuity = 3.980041e+00 @ (4, 11, 17) Rank#45, time step = 2, continuity = 2.594768e-03 @ (4, 50, 34) Rank#46, time step = 2, continuity = 2.060500e-03 @ (4, 50, 34) Rank#1, time step = 2, continuity = 2.221491e-04 @ (4, 4, 4) Rank#0, time step = 2, continuity = 2.750387e-08 @ (19, 51, 7) Rank#4, time step = 2, continuity = 6.838965e-03 @ (4, 51, 35) Rank#5, time step = 2, continuity = 2.685059e-03 @ (4, 4, 4) Rank#2, time step = 2, continuity = 3.305379e-03 @ (4, 51, 35) Rank#36, time step = 2, continuity = 1.145714e-02 @ (5, 51, 4) Rank#31, time step = 2, continuity = 2.043125e-03 @ (4, 50, 5) Rank#37, time step = 2, continuity = 2.641305e-03 @ (4, 4, 34) Rank#30, time step = 2, continuity = 2.021057e-03 @ (4, 50, 5) Rank#28, time step = 2, continuity = 2.169531e-02 @ (4, 4, 20) Rank#29, time step = 2, continuity = 2.251697e-03 @ (4, 50, 5) Rank#13, time step = 2, continuity = 2.641033e-03 @ (4, 50, 4) Rank#14, time step = 2, continuity = 2.080491e-03 @ (4, 50, 4) Rank#8, time step = 2, continuity = 3.839277e-05 @ (41, 4, 35) Rank#20, time step = 2, continuity = 2.088852e-02 @ (4, 51, 20) Rank#9, time step = 2, continuity = 2.025106e-04 @ (4, 50, 4) Rank#22, time step = 2, continuity = 2.042619e-03 @ (4, 4, 5) Rank#10, time step = 2, continuity = 3.293410e-03 @ (4, 5, 35) Rank#21, time step = 2, continuity = 2.318005e-03 @ (4, 4, 5) Rank#25, time step = 2, continuity = 5.028256e-03 @ (41, 4, 4) Rank#26, time step = 2, continuity = 8.660592e-01 @ (39, 4, 17) Rank#11, time step = 2, continuity = 4.892623e-02 @ (4, 4, 35) Rank#38, time step = 2, continuity = 2.080645e-03 @ (4, 4, 34) Rank#40, time step = 2, continuity = 2.570407e-04 @ (41, 5, 4) Rank#41, time step = 2, continuity = 4.631534e-03 @ (41, 5, 4) Rank#16, time step = 2, continuity = 3.189125e-04 @ (41, 51, 4) Rank#42, time step = 2, continuity = 1.119371e-01 @ (40, 5, 4) Rank#17, time step = 2, continuity = 4.813800e-03 @ (41, 51, 4) Rank#43, time step = 2, continuity = 1.055263e-01 @ (19, 5, 4) Rank#18, time step = 2, continuity = 9.377534e-02 @ (39, 51, 4) Rank#15, time step = 2, continuity = 2.050923e-03 @ (4, 50, 4) Rank#19, time step = 2, continuity = 4.025784e+00 @ (4, 51, 13) Rank#32, time step = 2, continuity = 2.570683e-04 @ (41, 51, 4) Rank#33, time step = 2, continuity = 4.597828e-03 @ (41, 51, 4) Rank#23, time step = 2, continuity = 2.050687e-03 @ (4, 4, 5) Rank#34, time step = 2, continuity = 1.111301e-01 @ (40, 51, 4) Rank#35, time step = 2, continuity = 1.047201e-01 @ (19, 51, 4) Rank#47, time step = 2, continuity = 2.043574e-03 @ (4, 50, 34) Rank#3, time step = 2, continuity = 4.698379e-02 @ (4, 51, 35) Rank#12, time step = 2, continuity = 7.182269e-03 @ (4, 4, 35) Rank#7, time step = 2, continuity = 2.058267e-03 @ (4, 4, 4) Rank#6, time step = 2, continuity = 2.100210e-03 @ (4, 4, 4) Rank#39, time step = 2, continuity = 2.051061e-03 @ (4, 5, 34) Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 Current time step = 3, Total time step number = 5 0 KSP unpreconditioned resid norm 1.498434584652e+01 true resid norm 1.498434584652e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.092790363921e+00 true resid norm 1.092790363921e+00 ||r(i)||/||b|| 7.292880017017e-02 2 KSP unpreconditioned resid norm 3.035607510875e-01 true resid norm 3.035607510875e-01 ||r(i)||/||b|| 2.025852541024e-02 3 KSP unpreconditioned resid norm 1.046104233532e-01 true resid norm 1.046104233532e-01 ||r(i)||/||b|| 6.981313994265e-03 4 KSP unpreconditioned resid norm 4.900330197785e-02 true resid norm 4.900330197786e-02 ||r(i)||/||b|| 3.270299716771e-03 5 KSP unpreconditioned resid norm 3.243734826670e-02 true resid norm 3.243734826671e-02 ||r(i)||/||b|| 2.164749038694e-03 6 KSP unpreconditioned resid norm 2.125601515562e-02 true resid norm 2.125601515562e-02 ||r(i)||/||b|| 1.418548088341e-03 7 KSP unpreconditioned resid norm 1.320366086225e-02 true resid norm 1.320366086225e-02 ||r(i)||/||b|| 8.811636488838e-04 8 KSP unpreconditioned resid norm 9.434930974812e-03 true resid norm 9.434930974811e-03 ||r(i)||/||b|| 6.296525101231e-04 9 KSP unpreconditioned resid norm 6.928273956785e-03 true resid norm 6.928273956791e-03 ||r(i)||/||b|| 4.623674618670e-04 10 KSP unpreconditioned resid norm 5.054559978185e-03 true resid norm 5.054559978187e-03 ||r(i)||/||b|| 3.373226986323e-04 11 KSP unpreconditioned resid norm 3.879257749830e-03 true resid norm 3.879257749830e-03 ||r(i)||/||b|| 2.588873608208e-04 12 KSP unpreconditioned resid norm 3.115677995555e-03 true resid norm 3.115677995557e-03 ||r(i)||/||b|| 2.079288630595e-04 13 KSP unpreconditioned resid norm 2.397098909417e-03 true resid norm 2.397098909416e-03 ||r(i)||/||b|| 1.599735439885e-04 14 KSP unpreconditioned resid norm 1.919635663198e-03 true resid norm 1.919635663196e-03 ||r(i)||/||b|| 1.281094071679e-04 15 KSP unpreconditioned resid norm 1.516959045641e-03 true resid norm 1.516959045638e-03 ||r(i)||/||b|| 1.012362542333e-04 16 KSP unpreconditioned resid norm 1.227746545064e-03 true resid norm 1.227746545063e-03 ||r(i)||/||b|| 8.193527816555e-05 17 KSP unpreconditioned resid norm 9.567761331953e-04 true resid norm 9.567761331946e-04 ||r(i)||/||b|| 6.385171184614e-05 18 KSP unpreconditioned resid norm 8.061097903490e-04 true resid norm 8.061097903493e-04 ||r(i)||/||b|| 5.379679557628e-05 19 KSP unpreconditioned resid norm 6.448576549729e-04 true resid norm 6.448576549744e-04 ||r(i)||/||b|| 4.303542253894e-05 20 KSP unpreconditioned resid norm 5.367404070636e-04 true resid norm 5.367404070649e-04 ||r(i)||/||b|| 3.582007600216e-05 21 KSP unpreconditioned resid norm 4.465369943436e-04 true resid norm 4.465369943470e-04 ||r(i)||/||b|| 2.980023278433e-05 22 KSP unpreconditioned resid norm 3.708072163845e-04 true resid norm 3.708072163881e-04 ||r(i)||/||b|| 2.474630659131e-05 23 KSP unpreconditioned resid norm 2.994136346519e-04 true resid norm 2.994136346554e-04 ||r(i)||/||b|| 1.998176214846e-05 24 KSP unpreconditioned resid norm 2.368415263321e-04 true resid norm 2.368415263359e-04 ||r(i)||/||b|| 1.580593031967e-05 25 KSP unpreconditioned resid norm 1.843154328279e-04 true resid norm 1.843154328311e-04 ||r(i)||/||b|| 1.230053248363e-05 26 KSP unpreconditioned resid norm 1.374799193076e-04 true resid norm 1.374799193111e-04 ||r(i)||/||b|| 9.174902976697e-06 27 KSP unpreconditioned resid norm 1.078590988123e-04 true resid norm 1.078590988147e-04 ||r(i)||/||b|| 7.198118617891e-06 28 KSP unpreconditioned resid norm 8.366418218981e-05 true resid norm 8.366418219349e-05 ||r(i)||/||b|| 5.583439080388e-06 29 KSP unpreconditioned resid norm 6.486198779907e-05 true resid norm 6.486198780324e-05 ||r(i)||/||b|| 4.328649943589e-06 30 KSP unpreconditioned resid norm 4.869596253714e-05 true resid norm 4.869596253714e-05 ||r(i)||/||b|| 3.249789015545e-06 31 KSP unpreconditioned resid norm 4.197253762996e-05 true resid norm 4.197253762997e-05 ||r(i)||/||b|| 2.801092424047e-06 32 KSP unpreconditioned resid norm 3.477255859637e-05 true resid norm 3.477255859636e-05 ||r(i)||/||b|| 2.320592367030e-06 33 KSP unpreconditioned resid norm 2.984747219315e-05 true resid norm 2.984747219315e-05 ||r(i)||/||b|| 1.991910257470e-06 34 KSP unpreconditioned resid norm 2.473689623528e-05 true resid norm 2.473689623530e-05 ||r(i)||/||b|| 1.650849258865e-06 35 KSP unpreconditioned resid norm 1.955260027410e-05 true resid norm 1.955260027416e-05 ||r(i)||/||b|| 1.304868459020e-06 36 KSP unpreconditioned resid norm 1.694685253795e-05 true resid norm 1.694685253801e-05 ||r(i)||/||b|| 1.130970461547e-06 37 KSP unpreconditioned resid norm 1.449937027782e-05 true resid norm 1.449937027791e-05 ||r(i)||/||b|| 9.676345184785e-07 38 KSP unpreconditioned resid norm 1.264545410891e-05 true resid norm 1.264545410903e-05 ||r(i)||/||b|| 8.439109880776e-07 39 KSP unpreconditioned resid norm 1.079008846191e-05 true resid norm 1.079008846223e-05 ||r(i)||/||b|| 7.200907248641e-07 40 KSP unpreconditioned resid norm 9.100895988825e-06 true resid norm 9.100895989285e-06 ||r(i)||/||b|| 6.073602466538e-07 41 KSP unpreconditioned resid norm 7.434748605604e-06 true resid norm 7.434748606064e-06 ||r(i)||/||b|| 4.961677127729e-07 42 KSP unpreconditioned resid norm 5.974987079856e-06 true resid norm 5.974987080279e-06 ||r(i)||/||b|| 3.987486101482e-07 43 KSP unpreconditioned resid norm 4.731598282761e-06 true resid norm 4.731598283153e-06 ||r(i)||/||b|| 3.157694257473e-07 44 KSP unpreconditioned resid norm 3.527588850234e-06 true resid norm 3.527588850504e-06 ||r(i)||/||b|| 2.354182749542e-07 45 KSP unpreconditioned resid norm 2.688764015030e-06 true resid norm 2.688764015219e-06 ||r(i)||/||b|| 1.794381978873e-07 46 KSP unpreconditioned resid norm 2.023673546681e-06 true resid norm 2.023673546778e-06 ||r(i)||/||b|| 1.350525119686e-07 47 KSP unpreconditioned resid norm 1.510010378674e-06 true resid norm 1.510010378681e-06 ||r(i)||/||b|| 1.007725258178e-07 48 KSP unpreconditioned resid norm 1.140779932135e-06 true resid norm 1.140779932032e-06 ||r(i)||/||b|| 7.613144702591e-08 Linear solve converged due to CONVERGED_RTOL iterations 48 Residual norm 1.14078e-06 0 KSP unpreconditioned resid norm 1.652982043410e+00 true resid norm 1.652982043410e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.040644076118e-01 true resid norm 1.040644076118e-01 ||r(i)||/||b|| 6.295555842646e-02 2 KSP unpreconditioned resid norm 2.849057515307e-02 true resid norm 2.849057515307e-02 ||r(i)||/||b|| 1.723586488229e-02 3 KSP unpreconditioned resid norm 1.053181684125e-02 true resid norm 1.053181684125e-02 ||r(i)||/||b|| 6.371404264939e-03 4 KSP unpreconditioned resid norm 4.749233649129e-03 true resid norm 4.749233649130e-03 ||r(i)||/||b|| 2.873130817158e-03 5 KSP unpreconditioned resid norm 3.048765576587e-03 true resid norm 3.048765576587e-03 ||r(i)||/||b|| 1.844403324732e-03 6 KSP unpreconditioned resid norm 1.972895444966e-03 true resid norm 1.972895444966e-03 ||r(i)||/||b|| 1.193537130565e-03 7 KSP unpreconditioned resid norm 1.216104509688e-03 true resid norm 1.216104509688e-03 ||r(i)||/||b|| 7.357034001285e-04 8 KSP unpreconditioned resid norm 8.397302896056e-04 true resid norm 8.397302896054e-04 ||r(i)||/||b|| 5.080093234850e-04 9 KSP unpreconditioned resid norm 6.264979217511e-04 true resid norm 6.264979217521e-04 ||r(i)||/||b|| 3.790107244358e-04 10 KSP unpreconditioned resid norm 4.502188878110e-04 true resid norm 4.502188878118e-04 ||r(i)||/||b|| 2.723676821577e-04 11 KSP unpreconditioned resid norm 3.486290906189e-04 true resid norm 3.486290906193e-04 ||r(i)||/||b|| 2.109091819898e-04 12 KSP unpreconditioned resid norm 2.892294177841e-04 true resid norm 2.892294177842e-04 ||r(i)||/||b|| 1.749743253034e-04 13 KSP unpreconditioned resid norm 2.455836801619e-04 true resid norm 2.455836801616e-04 ||r(i)||/||b|| 1.485700834687e-04 14 KSP unpreconditioned resid norm 2.080058928662e-04 true resid norm 2.080058928656e-04 ||r(i)||/||b|| 1.258367528522e-04 15 KSP unpreconditioned resid norm 1.769405745688e-04 true resid norm 1.769405745682e-04 ||r(i)||/||b|| 1.070432526921e-04 16 KSP unpreconditioned resid norm 1.532177881785e-04 true resid norm 1.532177881780e-04 ||r(i)||/||b|| 9.269174386308e-05 17 KSP unpreconditioned resid norm 1.213202871017e-04 true resid norm 1.213202871011e-04 ||r(i)||/||b|| 7.339480037595e-05 18 KSP unpreconditioned resid norm 1.008586917109e-04 true resid norm 1.008586917102e-04 ||r(i)||/||b|| 6.101620529536e-05 19 KSP unpreconditioned resid norm 7.536678882281e-05 true resid norm 7.536678882237e-05 ||r(i)||/||b|| 4.559443892500e-05 20 KSP unpreconditioned resid norm 6.165856591295e-05 true resid norm 6.165856591247e-05 ||r(i)||/||b|| 3.730141301795e-05 21 KSP unpreconditioned resid norm 4.781675773685e-05 true resid norm 4.781675773624e-05 ||r(i)||/||b|| 2.892757239976e-05 22 KSP unpreconditioned resid norm 4.111930662611e-05 true resid norm 4.111930662552e-05 ||r(i)||/||b|| 2.487583382376e-05 23 KSP unpreconditioned resid norm 3.231084932122e-05 true resid norm 3.231084932060e-05 ||r(i)||/||b|| 1.954700563712e-05 24 KSP unpreconditioned resid norm 2.629119845771e-05 true resid norm 2.629119845736e-05 ||r(i)||/||b|| 1.590531401244e-05 25 KSP unpreconditioned resid norm 2.062134326649e-05 true resid norm 2.062134326590e-05 ||r(i)||/||b|| 1.247523731314e-05 26 KSP unpreconditioned resid norm 1.598609489324e-05 true resid norm 1.598609489274e-05 ||r(i)||/||b|| 9.671063854853e-06 27 KSP unpreconditioned resid norm 1.293327139531e-05 true resid norm 1.293327139479e-05 ||r(i)||/||b|| 7.824205620597e-06 28 KSP unpreconditioned resid norm 1.035949175128e-05 true resid norm 1.035949175088e-05 ||r(i)||/||b|| 6.267153229026e-06 29 KSP unpreconditioned resid norm 8.880493935672e-06 true resid norm 8.880493935523e-06 ||r(i)||/||b|| 5.372407988900e-06 30 KSP unpreconditioned resid norm 7.137871934412e-06 true resid norm 7.137871934412e-06 ||r(i)||/||b|| 4.318178750259e-06 31 KSP unpreconditioned resid norm 6.707215155334e-06 true resid norm 6.707215155334e-06 ||r(i)||/||b|| 4.057645503213e-06 32 KSP unpreconditioned resid norm 5.556103817925e-06 true resid norm 5.556103817933e-06 ||r(i)||/||b|| 3.361260843748e-06 33 KSP unpreconditioned resid norm 4.988355482067e-06 true resid norm 4.988355482078e-06 ||r(i)||/||b|| 3.017791694693e-06 34 KSP unpreconditioned resid norm 4.087915820136e-06 true resid norm 4.087915820152e-06 ||r(i)||/||b|| 2.473055189226e-06 35 KSP unpreconditioned resid norm 3.481285629031e-06 true resid norm 3.481285629045e-06 ||r(i)||/||b|| 2.106063791149e-06 36 KSP unpreconditioned resid norm 2.775692176766e-06 true resid norm 2.775692176776e-06 ||r(i)||/||b|| 1.679202861182e-06 37 KSP unpreconditioned resid norm 2.326242471264e-06 true resid norm 2.326242471274e-06 ||r(i)||/||b|| 1.407300509130e-06 38 KSP unpreconditioned resid norm 1.903169493624e-06 true resid norm 1.903169493629e-06 ||r(i)||/||b|| 1.151355213577e-06 39 KSP unpreconditioned resid norm 1.647136962470e-06 true resid norm 1.647136962479e-06 ||r(i)||/||b|| 9.964639174670e-07 40 KSP unpreconditioned resid norm 1.341776814255e-06 true resid norm 1.341776814265e-06 ||r(i)||/||b|| 8.117310285457e-07 41 KSP unpreconditioned resid norm 1.113514152334e-06 true resid norm 1.113514152345e-06 ||r(i)||/||b|| 6.736395938384e-07 42 KSP unpreconditioned resid norm 8.915555405500e-07 true resid norm 8.915555405616e-07 ||r(i)||/||b|| 5.393619030020e-07 43 KSP unpreconditioned resid norm 7.237736566449e-07 true resid norm 7.237736566594e-07 ||r(i)||/||b|| 4.378593582096e-07 44 KSP unpreconditioned resid norm 5.921887748494e-07 true resid norm 5.921887748552e-07 ||r(i)||/||b|| 3.582548142106e-07 45 KSP unpreconditioned resid norm 4.832497255683e-07 true resid norm 4.832497255690e-07 ||r(i)||/||b|| 2.923502572189e-07 46 KSP unpreconditioned resid norm 4.072281865093e-07 true resid norm 4.072281865119e-07 ||r(i)||/||b|| 2.463597158453e-07 47 KSP unpreconditioned resid norm 3.355562061436e-07 true resid norm 3.355562061460e-07 ||r(i)||/||b|| 2.030005150291e-07 48 KSP unpreconditioned resid norm 2.849049830237e-07 true resid norm 2.849049830229e-07 ||r(i)||/||b|| 1.723581839009e-07 49 KSP unpreconditioned resid norm 2.266132667399e-07 true resid norm 2.266132667368e-07 ||r(i)||/||b|| 1.370936046403e-07 50 KSP unpreconditioned resid norm 1.739841395163e-07 true resid norm 1.739841395054e-07 ||r(i)||/||b|| 1.052547063043e-07 51 KSP unpreconditioned resid norm 1.268356186763e-07 true resid norm 1.268356186611e-07 ||r(i)||/||b|| 7.673139533900e-08 Linear solve converged due to CONVERGED_RTOL iterations 51 Residual norm 1.26836e-07 Rank#44, time step = 3, continuity = 1.368123e-02 @ (5, 5, 4) Rank#24, time step = 3, continuity = 3.885754e-04 @ (41, 4, 4) Rank#10, time step = 3, continuity = 4.105729e-03 @ (4, 5, 35) Rank#3, time step = 3, continuity = 5.609826e-02 @ (4, 51, 35) Rank#45, time step = 3, continuity = 1.929346e-03 @ (4, 50, 34) Rank#27, time step = 3, continuity = 3.546324e+00 @ (4, 11, 17) Rank#11, time step = 3, continuity = 5.897940e-02 @ (4, 4, 35) Rank#2, time step = 3, continuity = 4.119999e-03 @ (4, 51, 35) Rank#0, time step = 3, continuity = 1.960288e-08 @ (19, 51, 7) Rank#46, time step = 3, continuity = 1.202898e-03 @ (4, 50, 34) Rank#8, time step = 3, continuity = 4.222265e-05 @ (41, 4, 35) Rank#4, time step = 3, continuity = 9.702817e-03 @ (4, 51, 35) Rank#5, time step = 3, continuity = 2.042632e-03 @ (4, 4, 4) Rank#6, time step = 3, continuity = 1.251370e-03 @ (4, 4, 4) Rank#30, time step = 3, continuity = 1.158913e-03 @ (4, 50, 5) Rank#14, time step = 3, continuity = 1.228256e-03 @ (4, 50, 4) Rank#7, time step = 3, continuity = 1.121091e-03 @ (4, 4, 4) Rank#15, time step = 3, continuity = 1.112731e-03 @ (4, 50, 4) Rank#16, time step = 3, continuity = 3.469996e-04 @ (41, 51, 4) Rank#17, time step = 3, continuity = 5.357466e-03 @ (41, 51, 4) Rank#18, time step = 3, continuity = 1.080270e-01 @ (39, 51, 4) Rank#25, time step = 3, continuity = 5.597218e-03 @ (41, 4, 4) Rank#13, time step = 3, continuity = 1.989096e-03 @ (4, 50, 4) Rank#12, time step = 3, continuity = 1.013889e-02 @ (4, 4, 35) Rank#26, time step = 3, continuity = 7.539334e-01 @ (39, 4, 17) Rank#23, time step = 3, continuity = 1.112361e-03 @ (4, 4, 5) Rank#20, time step = 3, continuity = 2.731360e-02 @ (4, 51, 20) Rank#22, time step = 3, continuity = 1.184401e-03 @ (4, 4, 5) Rank#21, time step = 3, continuity = 1.597810e-03 @ (4, 4, 5) Rank#39, time step = 3, continuity = 1.112194e-03 @ (4, 5, 34) Rank#36, time step = 3, continuity = 1.367959e-02 @ (5, 51, 4) Rank#37, time step = 3, continuity = 1.986552e-03 @ (4, 4, 34) Rank#40, time step = 3, continuity = 2.850005e-04 @ (41, 5, 4) Rank#28, time step = 3, continuity = 2.828638e-02 @ (4, 4, 20) Rank#38, time step = 3, continuity = 1.226979e-03 @ (4, 4, 34) Rank#41, time step = 3, continuity = 5.161523e-03 @ (41, 5, 4) Rank#29, time step = 3, continuity = 1.517670e-03 @ (4, 50, 5) Rank#42, time step = 3, continuity = 1.294414e-01 @ (40, 5, 4) Rank#43, time step = 3, continuity = 1.224322e-01 @ (19, 5, 4) Rank#32, time step = 3, continuity = 2.852451e-04 @ (41, 51, 4) Rank#31, time step = 3, continuity = 1.103723e-03 @ (4, 50, 5) Rank#33, time step = 3, continuity = 5.164929e-03 @ (41, 51, 4) Rank#34, time step = 3, continuity = 1.287048e-01 @ (40, 51, 4) Rank#9, time step = 3, continuity = 2.683230e-04 @ (4, 50, 4) Rank#1, time step = 3, continuity = 2.941918e-04 @ (4, 4, 4) Rank#19, time step = 3, continuity = 3.689953e+00 @ (4, 51, 13) Rank#47, time step = 3, continuity = 1.103518e-03 @ (4, 50, 34) Rank#35, time step = 3, continuity = 1.214269e-01 @ (19, 51, 4) Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 Current time step = 4, Total time step number = 5 0 KSP unpreconditioned resid norm 1.800295456670e+01 true resid norm 1.800295456670e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 1.295938894634e+00 true resid norm 1.295938894634e+00 ||r(i)||/||b|| 7.198478948734e-02 2 KSP unpreconditioned resid norm 3.597174991333e-01 true resid norm 3.597174991333e-01 ||r(i)||/||b|| 1.998102576999e-02 3 KSP unpreconditioned resid norm 1.228849356110e-01 true resid norm 1.228849356110e-01 ||r(i)||/||b|| 6.825820459396e-03 4 KSP unpreconditioned resid norm 5.526977676103e-02 true resid norm 5.526977676104e-02 ||r(i)||/||b|| 3.070039229186e-03 5 KSP unpreconditioned resid norm 3.618819388233e-02 true resid norm 3.618819388233e-02 ||r(i)||/||b|| 2.010125268508e-03 6 KSP unpreconditioned resid norm 2.416631959864e-02 true resid norm 2.416631959864e-02 ||r(i)||/||b|| 1.342352973736e-03 7 KSP unpreconditioned resid norm 1.499930293446e-02 true resid norm 1.499930293447e-02 ||r(i)||/||b|| 8.331578507791e-04 8 KSP unpreconditioned resid norm 1.055050367432e-02 true resid norm 1.055050367432e-02 ||r(i)||/||b|| 5.860428984156e-04 9 KSP unpreconditioned resid norm 7.822204060520e-03 true resid norm 7.822204060528e-03 ||r(i)||/||b|| 4.344955730208e-04 10 KSP unpreconditioned resid norm 5.732827542959e-03 true resid norm 5.732827542962e-03 ||r(i)||/||b|| 3.184381497893e-04 11 KSP unpreconditioned resid norm 4.349201692653e-03 true resid norm 4.349201692655e-03 ||r(i)||/||b|| 2.415826622536e-04 12 KSP unpreconditioned resid norm 3.507456574416e-03 true resid norm 3.507456574419e-03 ||r(i)||/||b|| 1.948267192156e-04 13 KSP unpreconditioned resid norm 2.714359769664e-03 true resid norm 2.714359769663e-03 ||r(i)||/||b|| 1.507730167071e-04 14 KSP unpreconditioned resid norm 2.159613131461e-03 true resid norm 2.159613131459e-03 ||r(i)||/||b|| 1.199588169518e-04 15 KSP unpreconditioned resid norm 1.704473557682e-03 true resid norm 1.704473557679e-03 ||r(i)||/||b|| 9.467743482680e-05 16 KSP unpreconditioned resid norm 1.390545859097e-03 true resid norm 1.390545859096e-03 ||r(i)||/||b|| 7.723986937498e-05 17 KSP unpreconditioned resid norm 1.077180358743e-03 true resid norm 1.077180358741e-03 ||r(i)||/||b|| 5.983353203220e-05 18 KSP unpreconditioned resid norm 9.048346152026e-04 true resid norm 9.048346152024e-04 ||r(i)||/||b|| 5.026033987087e-05 19 KSP unpreconditioned resid norm 7.300819022225e-04 true resid norm 7.300819022251e-04 ||r(i)||/||b|| 4.055344913081e-05 20 KSP unpreconditioned resid norm 6.034455779373e-04 true resid norm 6.034455779385e-04 ||r(i)||/||b|| 3.351925239286e-05 21 KSP unpreconditioned resid norm 5.027864502361e-04 true resid norm 5.027864502383e-04 ||r(i)||/||b|| 2.792799639500e-05 22 KSP unpreconditioned resid norm 4.185448641713e-04 true resid norm 4.185448641737e-04 ||r(i)||/||b|| 2.324867635604e-05 23 KSP unpreconditioned resid norm 3.383127450993e-04 true resid norm 3.383127451025e-04 ||r(i)||/||b|| 1.879206792691e-05 24 KSP unpreconditioned resid norm 2.658271933588e-04 true resid norm 2.658271933632e-04 ||r(i)||/||b|| 1.476575371994e-05 25 KSP unpreconditioned resid norm 2.086016977258e-04 true resid norm 2.086016977289e-04 ||r(i)||/||b|| 1.158708127358e-05 26 KSP unpreconditioned resid norm 1.553878432255e-04 true resid norm 1.553878432301e-04 ||r(i)||/||b|| 8.631241202902e-06 27 KSP unpreconditioned resid norm 1.207652470263e-04 true resid norm 1.207652470298e-04 ||r(i)||/||b|| 6.708079308999e-06 28 KSP unpreconditioned resid norm 9.478447325293e-05 true resid norm 9.478447325797e-05 ||r(i)||/||b|| 5.264939868996e-06 29 KSP unpreconditioned resid norm 7.309077489139e-05 true resid norm 7.309077489702e-05 ||r(i)||/||b|| 4.059932197585e-06 30 KSP unpreconditioned resid norm 5.488099653620e-05 true resid norm 5.488099653620e-05 ||r(i)||/||b|| 3.048443872524e-06 31 KSP unpreconditioned resid norm 4.691702649096e-05 true resid norm 4.691702649093e-05 ||r(i)||/||b|| 2.606073704019e-06 32 KSP unpreconditioned resid norm 3.921717594840e-05 true resid norm 3.921717594834e-05 ||r(i)||/||b|| 2.178374433098e-06 33 KSP unpreconditioned resid norm 3.340909038588e-05 true resid norm 3.340909038575e-05 ||r(i)||/||b|| 1.855755968387e-06 34 KSP unpreconditioned resid norm 2.772109424893e-05 true resid norm 2.772109424866e-05 ||r(i)||/||b|| 1.539808043505e-06 35 KSP unpreconditioned resid norm 2.185592027217e-05 true resid norm 2.185592027141e-05 ||r(i)||/||b|| 1.214018520706e-06 36 KSP unpreconditioned resid norm 1.903747666486e-05 true resid norm 1.903747666396e-05 ||r(i)||/||b|| 1.057464017555e-06 37 KSP unpreconditioned resid norm 1.624600448157e-05 true resid norm 1.624600448058e-05 ||r(i)||/||b|| 9.024076809383e-07 38 KSP unpreconditioned resid norm 1.420940242615e-05 true resid norm 1.420940242508e-05 ||r(i)||/||b|| 7.892816910929e-07 39 KSP unpreconditioned resid norm 1.203584566250e-05 true resid norm 1.203584566145e-05 ||r(i)||/||b|| 6.685483550411e-07 40 KSP unpreconditioned resid norm 1.020862316387e-05 true resid norm 1.020862316287e-05 ||r(i)||/||b|| 5.670526537766e-07 41 KSP unpreconditioned resid norm 8.241595972505e-06 true resid norm 8.241595971594e-06 ||r(i)||/||b|| 4.577912998146e-07 42 KSP unpreconditioned resid norm 6.689625267539e-06 true resid norm 6.689625266635e-06 ||r(i)||/||b|| 3.715848552441e-07 43 KSP unpreconditioned resid norm 5.248454162379e-06 true resid norm 5.248454161620e-06 ||r(i)||/||b|| 2.915329337845e-07 44 KSP unpreconditioned resid norm 3.931049414091e-06 true resid norm 3.931049413544e-06 ||r(i)||/||b|| 2.183557925994e-07 45 KSP unpreconditioned resid norm 3.002868065889e-06 true resid norm 3.002868065505e-06 ||r(i)||/||b|| 1.667986248801e-07 46 KSP unpreconditioned resid norm 2.243200121528e-06 true resid norm 2.243200121288e-06 ||r(i)||/||b|| 1.246017765016e-07 47 KSP unpreconditioned resid norm 1.693164287854e-06 true resid norm 1.693164287781e-06 ||r(i)||/||b|| 9.404924516737e-08 Linear solve converged due to CONVERGED_RTOL iterations 47 Residual norm 1.69316e-06 0 KSP unpreconditioned resid norm 1.412673063450e+00 true resid norm 1.412673063450e+00 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 8.531572199357e-02 true resid norm 8.531572199358e-02 ||r(i)||/||b|| 6.039311161296e-02 2 KSP unpreconditioned resid norm 2.354905984408e-02 true resid norm 2.354905984408e-02 ||r(i)||/||b|| 1.666985833691e-02 3 KSP unpreconditioned resid norm 8.866561466319e-03 true resid norm 8.866561466321e-03 ||r(i)||/||b|| 6.276442650265e-03 4 KSP unpreconditioned resid norm 4.007731044735e-03 true resid norm 4.007731044736e-03 ||r(i)||/||b|| 2.836984117859e-03 5 KSP unpreconditioned resid norm 2.562270094262e-03 true resid norm 2.562270094262e-03 ||r(i)||/||b|| 1.813774298211e-03 6 KSP unpreconditioned resid norm 1.648374750802e-03 true resid norm 1.648374750802e-03 ||r(i)||/||b|| 1.166848008538e-03 7 KSP unpreconditioned resid norm 1.012675849200e-03 true resid norm 1.012675849200e-03 ||r(i)||/||b|| 7.168508237332e-04 8 KSP unpreconditioned resid norm 6.975172897121e-04 true resid norm 6.975172897118e-04 ||r(i)||/||b|| 4.937570537433e-04 9 KSP unpreconditioned resid norm 5.194036995358e-04 true resid norm 5.194036995367e-04 ||r(i)||/||b|| 3.676743848064e-04 10 KSP unpreconditioned resid norm 3.696092053889e-04 true resid norm 3.696092053896e-04 ||r(i)||/||b|| 2.616381772629e-04 11 KSP unpreconditioned resid norm 2.850523803892e-04 true resid norm 2.850523803895e-04 ||r(i)||/||b|| 2.017822720377e-04 12 KSP unpreconditioned resid norm 2.350800134299e-04 true resid norm 2.350800134300e-04 ||r(i)||/||b|| 1.664079393259e-04 13 KSP unpreconditioned resid norm 2.005287837193e-04 true resid norm 2.005287837191e-04 ||r(i)||/||b|| 1.419498884117e-04 14 KSP unpreconditioned resid norm 1.693423716447e-04 true resid norm 1.693423716441e-04 ||r(i)||/||b|| 1.198737174407e-04 15 KSP unpreconditioned resid norm 1.450875061234e-04 true resid norm 1.450875061229e-04 ||r(i)||/||b|| 1.027042348840e-04 16 KSP unpreconditioned resid norm 1.255170696894e-04 true resid norm 1.255170696889e-04 ||r(i)||/||b|| 8.885075601452e-05 17 KSP unpreconditioned resid norm 1.001150732534e-04 true resid norm 1.001150732528e-04 ||r(i)||/||b|| 7.086924486848e-05 18 KSP unpreconditioned resid norm 8.324107526331e-05 true resid norm 8.324107526263e-05 ||r(i)||/||b|| 5.892451510282e-05 19 KSP unpreconditioned resid norm 6.233954903979e-05 true resid norm 6.233954903940e-05 ||r(i)||/||b|| 4.412878722776e-05 20 KSP unpreconditioned resid norm 5.068242281259e-05 true resid norm 5.068242281210e-05 ||r(i)||/||b|| 3.587696553675e-05 21 KSP unpreconditioned resid norm 3.925070811210e-05 true resid norm 3.925070811142e-05 ||r(i)||/||b|| 2.778470767720e-05 22 KSP unpreconditioned resid norm 3.353801271807e-05 true resid norm 3.353801271731e-05 ||r(i)||/||b|| 2.374081702626e-05 23 KSP unpreconditioned resid norm 2.647759351203e-05 true resid norm 2.647759351127e-05 ||r(i)||/||b|| 1.874290251319e-05 24 KSP unpreconditioned resid norm 2.135260655706e-05 true resid norm 2.135260655663e-05 ||r(i)||/||b|| 1.511503766093e-05 25 KSP unpreconditioned resid norm 1.687070927407e-05 true resid norm 1.687070927336e-05 ||r(i)||/||b|| 1.194240175582e-05 26 KSP unpreconditioned resid norm 1.293694698390e-05 true resid norm 1.293694698317e-05 ||r(i)||/||b|| 9.157778482429e-06 27 KSP unpreconditioned resid norm 1.049854679096e-05 true resid norm 1.049854679022e-05 ||r(i)||/||b|| 7.431688946185e-06 28 KSP unpreconditioned resid norm 8.323105069998e-06 true resid norm 8.323105069154e-06 ||r(i)||/||b|| 5.891741893078e-06 29 KSP unpreconditioned resid norm 7.149131008665e-06 true resid norm 7.149131007929e-06 ||r(i)||/||b|| 5.060711634486e-06 30 KSP unpreconditioned resid norm 5.765627614985e-06 true resid norm 5.765627614985e-06 ||r(i)||/||b|| 4.081360198730e-06 31 KSP unpreconditioned resid norm 5.420427196404e-06 true resid norm 5.420427196403e-06 ||r(i)||/||b|| 3.837000461497e-06 32 KSP unpreconditioned resid norm 4.532576209396e-06 true resid norm 4.532576209394e-06 ||r(i)||/||b|| 3.208510395408e-06 33 KSP unpreconditioned resid norm 4.064657789569e-06 true resid norm 4.064657789567e-06 ||r(i)||/||b|| 2.877281300770e-06 34 KSP unpreconditioned resid norm 3.349298785897e-06 true resid norm 3.349298785894e-06 ||r(i)||/||b|| 2.370894492540e-06 35 KSP unpreconditioned resid norm 2.848140811778e-06 true resid norm 2.848140811775e-06 ||r(i)||/||b|| 2.016135852990e-06 36 KSP unpreconditioned resid norm 2.272096193713e-06 true resid norm 2.272096193716e-06 ||r(i)||/||b|| 1.608366615391e-06 37 KSP unpreconditioned resid norm 1.902087742756e-06 true resid norm 1.902087742762e-06 ||r(i)||/||b|| 1.346445821028e-06 38 KSP unpreconditioned resid norm 1.556881030367e-06 true resid norm 1.556881030374e-06 ||r(i)||/||b|| 1.102081628548e-06 39 KSP unpreconditioned resid norm 1.348642308105e-06 true resid norm 1.348642308109e-06 ||r(i)||/||b|| 9.546740452567e-07 40 KSP unpreconditioned resid norm 1.100324584644e-06 true resid norm 1.100324584650e-06 ||r(i)||/||b|| 7.788954239433e-07 41 KSP unpreconditioned resid norm 9.053973900964e-07 true resid norm 9.053973900970e-07 ||r(i)||/||b|| 6.409107765427e-07 42 KSP unpreconditioned resid norm 7.230581243700e-07 true resid norm 7.230581243733e-07 ||r(i)||/||b|| 5.118368453966e-07 43 KSP unpreconditioned resid norm 5.825287157867e-07 true resid norm 5.825287157901e-07 ||r(i)||/||b|| 4.123591868930e-07 44 KSP unpreconditioned resid norm 4.762644417313e-07 true resid norm 4.762644417360e-07 ||r(i)||/||b|| 3.371370588555e-07 45 KSP unpreconditioned resid norm 3.865941721477e-07 true resid norm 3.865941721540e-07 ||r(i)||/||b|| 2.736614593682e-07 46 KSP unpreconditioned resid norm 3.263216396424e-07 true resid norm 3.263216396447e-07 ||r(i)||/||b|| 2.309958674003e-07 47 KSP unpreconditioned resid norm 2.681743797359e-07 true resid norm 2.681743797427e-07 ||r(i)||/||b|| 1.898347088800e-07 48 KSP unpreconditioned resid norm 2.281833238064e-07 true resid norm 2.281833238130e-07 ||r(i)||/||b|| 1.615259253657e-07 49 KSP unpreconditioned resid norm 1.815350740360e-07 true resid norm 1.815350740409e-07 ||r(i)||/||b|| 1.285046616501e-07 50 KSP unpreconditioned resid norm 1.392510493034e-07 true resid norm 1.392510493056e-07 ||r(i)||/||b|| 9.857273626037e-08 Linear solve converged due to CONVERGED_RTOL iterations 50 Residual norm 1.39251e-07 Rank#47, time step = 4, continuity = 9.072241e-04 @ (4, 50, 34) Rank#32, time step = 4, continuity = 2.993468e-04 @ (41, 51, 4) Rank#34, time step = 4, continuity = 1.409515e-01 @ (40, 51, 4) Rank#0, time step = 4, continuity = 2.806779e-08 @ (41, 51, 35) Rank#9, time step = 4, continuity = 3.242195e-04 @ (4, 50, 4) Rank#1, time step = 4, continuity = 3.559030e-04 @ (4, 4, 4) Rank#10, time step = 4, continuity = 4.821606e-03 @ (4, 5, 35) Rank#27, time step = 4, continuity = 3.181760e+00 @ (4, 8, 14) Rank#36, time step = 4, continuity = 1.548400e-02 @ (5, 51, 4) Rank#37, time step = 4, continuity = 1.847557e-03 @ (4, 4, 34) Rank#8, time step = 4, continuity = 4.587898e-05 @ (41, 4, 35) Rank#44, time step = 4, continuity = 1.552645e-02 @ (5, 5, 4) Rank#16, time step = 4, continuity = 3.709613e-04 @ (41, 51, 4) Rank#4, time step = 4, continuity = 1.173989e-02 @ (4, 51, 35) Rank#45, time step = 4, continuity = 1.782619e-03 @ (4, 50, 34) Rank#18, time step = 4, continuity = 1.182176e-01 @ (39, 51, 4) Rank#5, time step = 4, continuity = 1.908359e-03 @ (4, 4, 4) Rank#46, time step = 4, continuity = 9.872820e-04 @ (4, 50, 34) Rank#6, time step = 4, continuity = 1.039245e-03 @ (4, 4, 4) Rank#12, time step = 4, continuity = 1.240915e-02 @ (4, 4, 35) Rank#25, time step = 4, continuity = 6.008582e-03 @ (41, 4, 4) Rank#13, time step = 4, continuity = 1.847262e-03 @ (4, 50, 4) Rank#14, time step = 4, continuity = 1.013285e-03 @ (4, 50, 4) Rank#24, time step = 4, continuity = 4.176380e-04 @ (41, 4, 4) Rank#40, time step = 4, continuity = 2.993544e-04 @ (41, 5, 4) Rank#42, time step = 4, continuity = 1.417756e-01 @ (40, 5, 4) Rank#28, time step = 4, continuity = 3.335993e-02 @ (4, 4, 20) Rank#43, time step = 4, continuity = 1.360899e-01 @ (19, 5, 4) Rank#29, time step = 4, continuity = 1.310806e-03 @ (4, 50, 5) Rank#33, time step = 4, continuity = 5.532869e-03 @ (41, 51, 4) Rank#23, time step = 4, continuity = 9.159324e-04 @ (4, 4, 5) Rank#2, time step = 4, continuity = 4.837666e-03 @ (4, 51, 35) Rank#20, time step = 4, continuity = 3.225439e-02 @ (4, 51, 20) Rank#21, time step = 4, continuity = 1.402715e-03 @ (4, 4, 5) Rank#22, time step = 4, continuity = 9.657753e-04 @ (4, 4, 5) Rank#26, time step = 4, continuity = 6.323174e-01 @ (39, 4, 17) Rank#19, time step = 4, continuity = 3.372144e+00 @ (4, 51, 13) Rank#30, time step = 4, continuity = 9.366603e-04 @ (4, 50, 5) Rank#41, time step = 4, continuity = 5.527059e-03 @ (41, 5, 4) Rank#38, time step = 4, continuity = 1.013971e-03 @ (4, 4, 34) Rank#11, time step = 4, continuity = 6.808021e-02 @ (4, 4, 35) Rank#31, time step = 4, continuity = 9.066095e-04 @ (4, 50, 5) Rank#17, time step = 4, continuity = 5.747098e-03 @ (41, 51, 4) Rank#39, time step = 4, continuity = 9.165216e-04 @ (4, 5, 34) Rank#3, time step = 4, continuity = 6.498223e-02 @ (4, 51, 35) Rank#7, time step = 4, continuity = 9.252253e-04 @ (4, 4, 4) Rank#15, time step = 4, continuity = 9.161437e-04 @ (4, 50, 4) Rank#35, time step = 4, continuity = 1.349255e-01 @ (19, 51, 4) ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex45 on a arch-linux2-c-debug named compute-8-8.local with 48 processors, by zlwei Fri Oct 5 14:00:22 2012 Using Petsc Development HG revision: 98bf11863c3be31b7c2af504314a500bc64d88c9 HG Date: Wed Aug 29 13:51:08 2012 -0500 Max Max/Min Avg Total Time (sec): 1.246e+02 1.00015 1.246e+02 Objects: 5.499e+03 1.00000 5.499e+03 Flops: 8.761e+09 1.03903 8.570e+09 4.114e+11 Flops/sec: 7.034e+07 1.03901 6.881e+07 3.303e+09 MPI Messages: 1.089e+05 2.25599 7.804e+04 3.746e+06 MPI Message Lengths: 4.049e+08 1.75931 4.110e+03 1.540e+10 MPI Reductions: 2.272e+04 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 4.0932e+01 32.9% 4.6734e+08 0.1% 1.037e+04 0.3% 1.167e+03 28.4% 3.200e+01 0.1% 1: DMMG Setup: 3.5440e+00 2.8% 0.0000e+00 0.0% 1.528e+03 0.0% 6.567e-01 0.0% 3.500e+01 0.2% 2: Pressure RHS Setup: 1.0369e+01 8.3% 1.4382e+10 3.5% 3.013e+05 8.0% 2.590e+02 6.3% 3.635e+03 16.0% 3: Pressure Solve: 2.8890e+01 23.2% 1.8356e+11 44.6% 1.513e+06 40.4% 1.171e+03 28.5% 7.453e+03 32.8% 4: Corrector RHS Setup: 9.9616e+00 8.0% 1.4382e+10 3.5% 3.013e+05 8.0% 2.590e+02 6.3% 3.635e+03 16.0% 5: Corrector Solve: 3.0856e+01 24.8% 1.9859e+11 48.3% 1.619e+06 43.2% 1.253e+03 30.5% 7.929e+03 34.9% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecNorm 10 1.0 5.0277e-02 2.3 1.17e+06 1.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 0 0 12 0 0 31 1100 VecSet 1 1.0 2.1939e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 10 1.0 2.4841e-03 1.6 1.17e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 12 0 0 0 22260 VecScatterBegin 11 1.0 5.4202e-03 3.0 0.00e+00 0.0 2.5e+03 1.2e+04 0.0e+00 0 0 0 0 0 0 0 24 1 0 0 VecScatterEnd 11 1.0 2.4676e-0213.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 10 1.0 4.8111e-02 1.7 7.56e+06 1.0 2.0e+03 1.2e+04 0.0e+00 0 0 0 0 0 0 76 19 1 0 7415 --- Event Stage 1: DMMG Setup --- Event Stage 2: Pressure RHS Setup KSPGMRESOrthog 150 1.0 4.9527e-01 1.7 7.04e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 1 0 0 1 4 23 0 0 4 6736 KSPSetUp 40 1.0 2.1560e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+02 0 0 0 0 1 2 0 0 0 5 0 VecMDot 150 1.0 4.3920e-01 1.9 3.52e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 3 12 0 0 4 3798 VecNorm 165 1.0 3.3326e-01 2.4 7.04e+06 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 2 0 0 5 1001 VecScale 165 1.0 1.0014e-02 2.8 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 16657 VecCopy 15 1.0 1.6418e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 235 1.0 7.2906e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 15 1.0 1.8489e-03 2.1 6.40e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16402 VecMAXPY 165 1.0 9.6994e-02 2.0 4.16e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 14 0 0 0 20323 VecAssemblyBegin 215 1.0 6.1984e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.3e+02 0 0 0 0 3 5 0 0 0 17 0 VecAssemblyEnd 215 1.0 4.8971e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 165 1.0 3.1083e-02 1.8 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 5366 VecScatterBegin 375 1.0 6.6681e-02 3.3 0.00e+00 0.0 1.7e+05 3.4e+03 0.0e+00 0 0 5 4 0 0 0 56 60 0 0 VecScatterEnd 375 1.0 5.2347e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0 VecSetRandom 15 1.0 1.0735e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 165 1.0 3.4388e-01 2.3 1.06e+07 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 3 0 0 5 1455 MatMult 150 1.0 3.7738e-01 2.0 4.51e+07 1.0 5.7e+04 2.8e+03 0.0e+00 0 1 2 1 0 3 15 19 16 0 5631 MatConvert 15 1.0 6.1136e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatScale 15 1.0 1.3903e-01 4.2 4.83e+06 1.0 5.7e+03 2.8e+03 0.0e+00 0 0 0 0 0 1 2 2 2 0 1638 MatAssemblyBegin 195 1.0 1.5021e+00 1.4 0.00e+00 0.0 1.3e+04 6.0e+02 2.2e+02 1 0 0 0 1 13 0 4 1 6 0 MatAssemblyEnd 195 1.0 1.0008e+00 1.3 0.00e+00 0.0 5.2e+04 6.3e+02 6.0e+02 1 0 1 0 3 8 0 17 3 17 0 MatGetRow 639730 1.0 1.6904e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 MatCoarsen 15 1.0 6.1620e-01 1.0 0.00e+00 0.0 5.3e+04 3.7e+03 4.3e+02 0 0 1 1 2 6 0 18 20 12 0 MatPtAP 15 1.0 7.3923e-01 1.1 1.40e+07 1.1 4.8e+04 1.0e+03 4.0e+02 1 0 1 0 2 7 5 16 5 11 886 MatPtAPSymbolic 15 1.0 5.5060e-01 1.1 0.00e+00 0.0 4.4e+04 9.0e+02 3.8e+02 0 0 1 0 2 5 0 15 4 10 0 MatPtAPNumeric 15 1.0 2.1724e-01 1.3 1.40e+07 1.1 4.3e+03 2.2e+03 3.0e+01 0 0 0 0 0 2 5 1 1 1 3014 MatTrnMatMult 15 1.0 3.4586e+00 1.0 1.51e+08 1.1 3.9e+04 7.8e+03 4.4e+02 3 2 1 2 2 33 49 13 31 12 2035 MatGetLocalMat 45 1.0 1.1697e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 15 1.0 1.0516e-01 2.3 0.00e+00 0.0 1.7e+04 1.8e+03 3.0e+01 0 0 0 0 0 1 0 6 3 1 0 MatGetSymTrans 30 1.0 5.6214e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.6096e+00 1.0 3.07e+08 1.0 2.9e+05 3.3e+03 3.2e+03 7 3 8 6 14 83100 97 98 89 1670 PCGAMGgraph_AGG 3 1.0 3.7921e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 120 PCGAMGcoarse_AGG 3 1.0 1.0288e+00 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 10 10 9 13 6 1368 PCGAMGProl_AGG 3 1.0 1.2415e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.9839e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 153 PCGAMGcoarse_AGG 3 1.0 8.6779e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1622 PCGAMGProl_AGG 3 1.0 1.2736e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7482e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 166 PCGAMGcoarse_AGG 3 1.0 8.9944e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1565 PCGAMGProl_AGG 3 1.0 1.2325e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.5020e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.8151e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 162 PCGAMGcoarse_AGG 3 1.0 8.8194e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1596 PCGAMGProl_AGG 3 1.0 1.3077e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7213e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 167 PCGAMGcoarse_AGG 3 1.0 8.9080e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1580 PCGAMGProl_AGG 3 1.0 1.3203e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 5.9605e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 3: Pressure Solve KSPGMRESOrthog 1928 1.0 7.4579e+00 1.6 8.71e+08 1.0 0.0e+00 0.0e+00 1.9e+03 5 10 0 0 8 20 22 0 0 26 5531 KSPSetUp 20 1.0 2.8610e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 2.8903e+01 1.0 3.92e+09 1.0 1.5e+06 2.9e+03 7.5e+03 23 45 40 28 33 100100100100100 6351 VecMDot 1928 1.0 6.7700e+00 1.7 4.36e+08 1.0 0.0e+00 0.0e+00 1.9e+03 4 5 0 0 8 17 11 0 0 26 3046 VecNorm 3876 1.0 9.7277e+00 1.8 1.82e+08 1.0 0.0e+00 0.0e+00 3.9e+03 6 2 0 0 17 24 5 0 0 52 887 VecScale 3625 1.0 2.0565e-01 2.6 7.66e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 17592 VecCopy 1938 1.0 2.5364e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 7958 1.0 4.1689e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 2656 1.0 2.5585e-01 1.5 1.21e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 3 0 0 0 22458 VecAYPX 969 1.0 1.5139e-01 1.5 2.98e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 9321 VecWAXPY 5 1.0 2.1451e-03 1.6 2.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6445 VecMAXPY 3861 1.0 2.1212e+00 1.6 8.82e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 10 0 0 0 6 23 0 0 0 19683 VecAssemblyBegin 5 1.0 7.7687e-02 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 1.7405e-05 3.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 5071 1.0 6.4226e-01 2.6 0.00e+00 0.0 1.5e+06 2.9e+03 0.0e+00 0 0 40 28 0 1 0100100 0 0 VecScatterEnd 5071 1.0 4.6577e+00 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 10 0 0 0 0 0 VecNormalize 3374 1.0 7.3255e+00 1.7 1.86e+08 1.0 0.0e+00 0.0e+00 3.4e+03 4 2 0 0 15 18 5 0 0 45 1197 MatMult 3625 1.0 9.2039e+00 1.6 1.24e+09 1.0 1.2e+06 3.5e+03 0.0e+00 6 14 32 27 0 27 32 79 96 0 6359 MatMultAdd 723 1.0 1.2736e+00 3.1 3.08e+07 1.0 1.6e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1148 MatMultTranspose 723 1.0 9.4210e-01 1.9 3.08e+07 1.0 1.6e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1552 MatSolve 3374 1.2 3.9078e+00 1.6 8.80e+08 1.1 0.0e+00 0.0e+00 0.0e+00 3 10 0 0 0 11 22 0 0 0 10213 MatLUFactorSym 5 1.0 6.0294e-0367.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 20 1.0 9.0770e-02 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4485 MatILUFactorSym 15 1.0 5.6968e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 20 1.3 4.4918e-0458.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 20 1.3 6.2945e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 20 1.0 1.5481e-01 1.6 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2630 PCSetUpOnBlocks 1687 1.0 1.5688e-01 1.5 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2595 PCApply 241 1.0 1.9936e+01 1.0 2.30e+09 1.1 1.4e+06 2.3e+03 5.1e+03 16 26 38 21 23 68 58 94 73 69 5361 --- Event Stage 4: Corrector RHS Setup KSPGMRESOrthog 150 1.0 4.4386e-01 1.5 7.04e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 1 0 0 1 4 23 0 0 4 7516 KSPSetUp 40 1.0 1.8739e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+02 0 0 0 0 1 2 0 0 0 5 0 VecMDot 150 1.0 3.9025e-01 1.7 3.52e+07 1.0 0.0e+00 0.0e+00 1.5e+02 0 0 0 0 1 3 12 0 0 4 4274 VecNorm 165 1.0 3.0658e-01 2.2 7.04e+06 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 2 0 0 5 1088 VecScale 165 1.0 1.0942e-02 3.1 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 15243 VecCopy 15 1.0 3.6764e-03 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 235 1.0 7.4570e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 15 1.0 1.7416e-03 1.9 6.40e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 17413 VecMAXPY 165 1.0 1.0165e-01 2.1 4.16e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 14 0 0 0 19393 VecAssemblyBegin 215 1.0 5.6306e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.3e+02 0 0 0 0 3 5 0 0 0 17 0 VecAssemblyEnd 215 1.0 1.0600e-03 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 165 1.0 3.2980e-02 2.0 3.52e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 5057 VecScatterBegin 375 1.0 6.3563e-02 3.1 0.00e+00 0.0 1.7e+05 3.4e+03 0.0e+00 0 0 5 4 0 0 0 56 60 0 0 VecScatterEnd 375 1.0 5.1996e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0 VecSetRandom 15 1.0 9.5236e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 165 1.0 3.1596e-01 2.2 1.06e+07 1.0 0.0e+00 0.0e+00 1.6e+02 0 0 0 0 1 2 3 0 0 5 1584 MatMult 150 1.0 3.3380e-01 1.8 4.51e+07 1.0 5.7e+04 2.8e+03 0.0e+00 0 1 2 1 0 3 15 19 16 0 6366 MatConvert 15 1.0 5.5341e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatScale 15 1.0 9.3830e-02 3.1 4.83e+06 1.0 5.7e+03 2.8e+03 0.0e+00 0 0 0 0 0 1 2 2 2 0 2426 MatAssemblyBegin 195 1.0 1.3034e+00 1.5 0.00e+00 0.0 1.3e+04 6.0e+02 2.2e+02 1 0 0 0 1 11 0 4 1 6 0 MatAssemblyEnd 195 1.0 9.7578e-01 1.2 0.00e+00 0.0 5.2e+04 6.3e+02 6.0e+02 1 0 1 0 3 9 0 17 3 17 0 MatGetRow 639730 1.0 1.8638e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 MatCoarsen 15 1.0 6.0700e-01 1.1 0.00e+00 0.0 5.3e+04 3.7e+03 4.3e+02 0 0 1 1 2 6 0 18 20 12 0 MatPtAP 15 1.0 7.4058e-01 1.1 1.40e+07 1.1 4.8e+04 1.0e+03 4.0e+02 1 0 1 0 2 7 5 16 5 11 884 MatPtAPSymbolic 15 1.0 5.6122e-01 1.1 0.00e+00 0.0 4.4e+04 9.0e+02 3.8e+02 0 0 1 0 2 5 0 15 4 10 0 MatPtAPNumeric 15 1.0 2.1232e-01 1.3 1.40e+07 1.1 4.3e+03 2.2e+03 3.0e+01 0 0 0 0 0 2 5 1 1 1 3084 MatTrnMatMult 15 1.0 3.2677e+00 1.0 1.51e+08 1.1 3.9e+04 7.8e+03 4.4e+02 3 2 1 2 2 33 49 13 31 12 2154 MatGetLocalMat 45 1.0 1.1687e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01 0 0 0 0 0 1 0 0 0 2 0 MatGetBrAoCol 15 1.0 1.0224e-01 2.1 0.00e+00 0.0 1.7e+04 1.8e+03 3.0e+01 0 0 0 0 0 1 0 6 3 1 0 MatGetSymTrans 30 1.0 5.1825e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 5 1.0 8.2378e+00 1.0 3.07e+08 1.0 2.9e+05 3.3e+03 3.2e+03 7 3 8 6 14 83100 97 98 89 1746 PCGAMGgraph_AGG 3 1.0 2.6251e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 2 0 1 0 2 173 PCGAMGcoarse_AGG 3 1.0 8.3389e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 8 10 9 13 6 1688 PCGAMGProl_AGG 3 1.0 1.2837e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7516e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 165 PCGAMGcoarse_AGG 3 1.0 8.5538e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1646 PCGAMGProl_AGG 3 1.0 1.3015e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.3113e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.8946e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 157 PCGAMGcoarse_AGG 3 1.0 8.7704e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1605 PCGAMGProl_AGG 3 1.0 1.3723e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.1921e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7662e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 165 PCGAMGcoarse_AGG 3 1.0 9.1107e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1545 PCGAMGProl_AGG 3 1.0 1.2597e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCGAMGgraph_AGG 3 1.0 2.7047e-01 1.1 9.66e+05 1.0 3.4e+03 1.4e+03 5.7e+01 0 0 0 0 0 3 0 1 0 2 168 PCGAMGcoarse_AGG 3 1.0 8.7131e-01 1.0 3.03e+07 1.1 2.6e+04 5.0e+03 2.3e+02 1 0 1 1 1 9 10 9 13 6 1615 PCGAMGProl_AGG 3 1.0 1.3009e-01 1.1 0.00e+00 0.0 7.4e+03 2.0e+03 8.4e+01 0 0 0 0 0 1 0 2 2 2 0 PCGAMGPOpt_AGG 3 1.0 1.4067e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 --- Event Stage 5: Corrector Solve KSPGMRESOrthog 2064 1.0 7.7906e+00 1.5 9.62e+08 1.0 0.0e+00 0.0e+00 2.1e+03 5 11 0 0 9 20 23 0 0 26 5847 KSPSetUp 20 1.0 3.0756e-05 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 5 1.0 3.0870e+01 1.0 4.24e+09 1.0 1.6e+06 2.9e+03 7.9e+03 25 48 43 30 35 100100100100100 6433 VecMDot 2064 1.0 7.0200e+00 1.6 4.81e+08 1.0 0.0e+00 0.0e+00 2.1e+03 4 6 0 0 9 17 11 0 0 26 3245 VecNorm 4148 1.0 1.0233e+01 1.8 1.95e+08 1.0 0.0e+00 0.0e+00 4.1e+03 6 2 0 0 18 24 5 0 0 52 901 VecScale 3880 1.0 2.1841e-01 2.6 8.19e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 2 0 0 0 17724 VecCopy 2074 1.0 2.7166e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecSet 8519 1.0 4.5739e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 VecAXPY 2843 1.0 2.7114e-01 1.4 1.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 3 0 0 0 22680 VecAYPX 1037 1.0 1.6162e-01 1.5 3.19e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 9340 VecWAXPY 5 1.0 2.2409e-03 1.7 2.92e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6169 VecMAXPY 4133 1.0 2.3573e+00 1.6 9.75e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 11 0 0 0 6 23 0 0 0 19578 VecAssemblyBegin 5 1.0 3.8697e-02 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 VecAssemblyEnd 5 1.0 1.5497e-05 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecScatterBegin 5428 1.0 6.7515e-01 2.6 0.00e+00 0.0 1.6e+06 2.9e+03 0.0e+00 0 0 43 30 0 1 0100100 0 0 VecScatterEnd 5428 1.0 4.9356e+00 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 10 0 0 0 0 0 VecNormalize 3612 1.0 7.7312e+00 1.7 1.99e+08 1.0 0.0e+00 0.0e+00 3.6e+03 5 2 0 0 16 18 5 0 0 46 1215 MatMult 3880 1.0 9.8883e+00 1.5 1.33e+09 1.0 1.3e+06 3.5e+03 0.0e+00 7 15 34 29 0 27 32 79 96 0 6334 MatMultAdd 774 1.0 1.3812e+00 3.1 3.30e+07 1.0 1.7e+05 5.0e+02 0.0e+00 1 0 4 1 0 3 1 10 2 0 1133 MatMultTranspose 774 1.0 1.0163e+00 1.9 3.30e+07 1.0 1.7e+05 5.0e+02 0.0e+00 1 0 4 1 0 2 1 10 2 0 1540 MatSolve 3612 1.2 4.1856e+00 1.6 9.42e+08 1.1 0.0e+00 0.0e+00 0.0e+00 3 10 0 0 0 11 22 0 0 0 10208 MatLUFactorSym 5 1.0 5.9757e-0368.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 20 1.0 9.2276e-02 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4412 MatILUFactorSym 15 1.0 4.9916e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 20 1.3 4.3893e-0476.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 20 1.3 2.0333e-02 6.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 20 1.0 1.3867e-01 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2936 PCSetUpOnBlocks 1806 1.0 1.4069e-01 1.4 1.98e+07 2.5 0.0e+00 0.0e+00 7.0e+01 0 0 0 0 0 0 0 0 0 1 2894 PCApply 258 1.0 2.1168e+01 1.0 2.46e+09 1.1 1.5e+06 2.3e+03 5.5e+03 17 28 40 22 24 68 58 94 73 69 5404 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Container 0 10 5480 0 Krylov Solver 0 91 970112 0 Vector 11 1172 387333592 0 Vector Scatter 0 114 118104 0 Matrix 0 310 190831360 0 Distributed Mesh 0 22 16761112 0 Bipartite Graph 0 44 30096 0 Index Set 0 140 2717000 0 IS L to G Mapping 0 33 11117700 0 Preconditioner 0 90 89240 0 --- Event Stage 1: DMMG Setup Krylov Solver 1 0 0 0 Vector 5 4 5856 0 Vector Scatter 4 0 0 0 Distributed Mesh 2 0 0 0 Bipartite Graph 4 0 0 0 Index Set 10 10 511452 0 IS L to G Mapping 3 0 0 0 --- Event Stage 2: Pressure RHS Setup Container 5 0 0 0 Krylov Solver 60 15 451800 0 Vector 900 610 65595960 0 Vector Scatter 115 60 62160 0 Matrix 355 220 226589740 0 Matrix Coarsen 15 15 9180 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 280 270 3224240 0 IS L to G Mapping 15 0 0 0 Preconditioner 60 15 12360 0 Viewer 1 0 0 0 PetscRandom 15 15 9120 0 --- Event Stage 3: Pressure Solve Vector 782 492 230472480 0 Matrix 20 0 0 0 Index Set 60 0 0 0 --- Event Stage 4: Corrector RHS Setup Container 5 0 0 0 Krylov Solver 60 15 451800 0 Vector 900 610 65595960 0 Vector Scatter 115 60 62160 0 Matrix 355 220 226589740 0 Matrix Coarsen 15 15 9180 0 Distributed Mesh 10 0 0 0 Bipartite Graph 20 0 0 0 Index Set 280 270 3224240 0 IS L to G Mapping 15 0 0 0 Preconditioner 60 15 12360 0 PetscRandom 15 15 9120 0 --- Event Stage 5: Corrector Solve Vector 816 526 246399440 0 Matrix 20 0 0 0 Index Set 60 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000296211 Average time for zero size MPI_Send(): 0.000285755 #PETSc Option Table entries: -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1.0e-7 -ksp_type fgmres -log_summary -mg_levels_ksp_max_it 1 -mg_levels_ksp_type gmres -mg_levels_pc_type bjacobi -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Tue Sep 18 16:22:39 2012 Configure options: --with-debugging=0 --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich ----------------------------------------- Libraries compiled on Tue Sep 18 16:22:39 2012 on firefox.bioinfo.ittc.ku.edu Machine characteristics: Linux-2.6.18-92.1.13.el5-x86_64-with-redhat-5.2-Final Using PETSc directory: /bio/work1/zlwei/PETSc/petsc-dev Using PETSc arch: arch-linux2-c-debug ----------------------------------------- Using C compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include ----------------------------------------- Using C linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc Using Fortran linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lgcc_s -ldl ----------------------------------------- -------------- next part -------------- NumRank = 80 node = 0 mx = 600 my = 192 mz = 192 mm = 10 nn = 2 pp = 4 rank 12 Cylinder: neighbor = left: 218 right: 180 bottom: 96 top: 117 aft: 73 front: 48 rank = 12, NbrRanks are: L: 11, R: 13, B:2, T:22, A:-8, F:32!! rank = 12: xs = 120, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 12: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 12: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 13 Cylinder: neighbor = left: 218 right: 240 bottom: 96 top: 117 aft: 73 front: 48 rank = 13, NbrRanks are: L: 12, R: 14, B:3, T:23, A:-7, F:33!! rank = 13: xs = 180, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 13: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 13: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 2 Cylinder: neighbor = left: 218 right: 180 bottom: 73 top: 96 aft: 73 front: 48 rank = 2, NbrRanks are: L: 1, R: 3, B:-8, T:12, A:-18, F:22!! rank = 2: xs = 120, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 2: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 2: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 75 Cylinder: neighbor = left: 300 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 75, NbrRanks are: L: 74, R: 76, B:65, T:85, A:55, F:95!! rank = 75: xs = 300, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 75: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 75: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 45 Cylinder: neighbor = left: 300 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank = 45, NbrRanks are: L: 44, R: 46, B:35, T:55, A:25, F:65!! rank = 45: xs = 300, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 45: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 45: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 4 Cylinder: neighbor = left: 240 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 4, NbrRanks are: L: 3, R: 5, B:-6, T:14, A:-16, F:24!! rank = 4: xs = 240, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 4: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 4: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 10 Cylinder: neighbor = left: 218 right: 60 bottom: 96 top: 117 aft: 73 front: 48 rank 30 Cylinder: neighbor = left: 218 right: 60 bottom: 96 top: 117 aft: 73 front: 96 rank = 30, NbrRanks are: L: 29, R: 31, B:20, T:40, A:10, F:50!! rank = 30: xs = 0, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 30: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 30: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 34 Cylinder: neighbor = left: 240 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank 65 Cylinder: neighbor = left: 300 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 65, NbrRanks are: L: 64, R: 66, B:55, T:75, A:45, F:85!! rank = 65: xs = 300, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 65: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 65: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 6 Cylinder: neighbor = left: 360 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 6, NbrRanks are: L: 5, R: 7, B:-4, T:16, A:-14, F:26!! rank 24 Cylinder: neighbor = left: 240 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 24, NbrRanks are: L: 23, R: 25, B:14, T:34, A:4, F:44!! rank 32 Cylinder: neighbor = left: 218 right: 180 bottom: 96 top: 117 aft: 73 front: 96 rank = 32, NbrRanks are: L: 31, R: 33, B:22, T:42, A:12, F:52!! rank = 32: xs = 120, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 32: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 32: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 54 Cylinder: neighbor = left: 240 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 54, NbrRanks are: L: 53, R: 55, B:44, T:64, A:34, F:74!! rank = 54: xs = 240, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 54: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 54: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 6: xs = 360, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 6: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 6: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 24: xs = 240, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 24: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 24: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 33 Cylinder: neighbor = left: 218 right: 240 bottom: 96 top: 117 aft: 73 front: 96 rank = 33, NbrRanks are: L: 32, R: 34, B:23, T:43, A:13, F:53!! rank = 33: xs = 180, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 33: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 33: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 53 Cylinder: neighbor = left: 218 right: 240 bottom: 96 top: 117 aft: 96 front: 117 rank = 53, NbrRanks are: L: 52, R: 54, B:43, T:63, A:33, F:73!! rank = 53: xs = 180, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 53: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 53: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 3 Cylinder: neighbor = left: 218 right: 240 bottom: 73 top: 96 aft: 73 front: 48 rank = 3, NbrRanks are: L: 2, R: 4, B:-7, T:13, A:-17, F:23!! rank 31 Cylinder: neighbor = left: 218 right: 120 bottom: 96 top: 117 aft: 73 front: 96 rank = 31, NbrRanks are: L: 30, R: 32, B:21, T:41, A:11, F:51!! rank = 34, NbrRanks are: L: 33, R: 35, B:24, T:44, A:14, F:54!! rank = 34: xs = 240, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 34: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 34: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 3: xs = 180, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 3: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 3: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 31: xs = 60, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 31: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 31: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 35 Cylinder: neighbor = left: 300 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank = 35, NbrRanks are: L: 34, R: 36, B:25, T:45, A:15, F:55!! rank = 35: xs = 300, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 35: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 35: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 5 Cylinder: neighbor = left: 300 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 5, NbrRanks are: L: 4, R: 6, B:-5, T:15, A:-15, F:25!! rank 25 Cylinder: neighbor = left: 300 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 25, NbrRanks are: L: 24, R: 26, B:15, T:35, A:5, F:45!! rank = 25: xs = 300, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 25: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 25: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 5: xs = 300, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 5: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 5: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 7 Cylinder: neighbor = left: 420 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 7, NbrRanks are: L: 6, R: 8, B:-3, T:17, A:-13, F:27!! rank = 7: xs = 420, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 7: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 7: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 0 Cylinder: neighbor = left: 218 right: 60 bottom: 73 top: 96 aft: 73 front: 48 rank = 0, NbrRanks are: L: -1, R: 1, B:-10, T:10, A:-20, F:20!! rank = 0: xs = 0, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 0: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 0: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 1 Cylinder: neighbor = left: 218 right: 120 bottom: 73 top: 96 aft: 73 front: 48 rank = 1, NbrRanks are: L: 0, R: 2, B:-9, T:11, A:-19, F:21!! rank = 1: xs = 60, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 1: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 1: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 8 Cylinder: neighbor = left: 480 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 8, NbrRanks are: L: 7, R: 9, B:-2, T:18, A:-12, F:28!! rank = 8: xs = 480, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 8: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 8: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 9 Cylinder: neighbor = left: 540 right: 262 bottom: 73 top: 96 aft: 73 front: 48 rank = 9, NbrRanks are: L: 8, R: 10, B:-1, T:19, A:-11, F:29!! rank = 9: xs = 540, ys = 0, zs = 0, mx = 600, my = 192, mz = 192! rank = 9: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0 rank = 9: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 10, NbrRanks are: L: 9, R: 11, B:0, T:20, A:-10, F:30!! rank = 10: xs = 0, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 10: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 10: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 14 Cylinder: neighbor = left: 240 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 14, NbrRanks are: L: 13, R: 15, B:4, T:24, A:-6, F:34!! rank = 14: xs = 240, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 14: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 14: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 15 Cylinder: neighbor = left: 300 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 15, NbrRanks are: L: 14, R: 16, B:5, T:25, A:-5, F:35!! rank = 15: xs = 300, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 15: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 15: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 44 Cylinder: neighbor = left: 240 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank = 44, NbrRanks are: L: 43, R: 45, B:34, T:54, A:24, F:64!! rank = 44: xs = 240, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 44: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 44: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 16 Cylinder: neighbor = left: 360 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 16, NbrRanks are: L: 15, R: 17, B:6, T:26, A:-4, F:36!! rank = 16: xs = 360, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 16: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 16: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 17 Cylinder: neighbor = left: 420 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 17, NbrRanks are: L: 16, R: 18, B:7, T:27, A:-3, F:37!! rank = 17: xs = 420, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 17: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 17: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 18 Cylinder: neighbor = left: 480 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 18, NbrRanks are: L: 17, R: 19, B:8, T:28, A:-2, F:38!! rank = 18: xs = 480, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 18: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 18: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 19 Cylinder: neighbor = left: 540 right: 262 bottom: 96 top: 117 aft: 73 front: 48 rank = 19, NbrRanks are: L: 18, R: 20, B:9, T:29, A:-1, F:39!! rank = 19: xs = 540, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 19: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 19: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 20 Cylinder: neighbor = left: 218 right: 60 bottom: 73 top: 96 aft: 73 front: 96 rank = 20, NbrRanks are: L: 19, R: 21, B:10, T:30, A:0, F:40!! rank = 20: xs = 0, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 20: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 20: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 21 Cylinder: neighbor = left: 218 right: 120 bottom: 73 top: 96 aft: 73 front: 96 rank = 21, NbrRanks are: L: 20, R: 22, B:11, T:31, A:1, F:41!! rank = 21: xs = 60, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 21: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 21: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 22 Cylinder: neighbor = left: 218 right: 180 bottom: 73 top: 96 aft: 73 front: 96 rank = 22, NbrRanks are: L: 21, R: 23, B:12, T:32, A:2, F:42!! rank = 22: xs = 120, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 22: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 22: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 23 Cylinder: neighbor = left: 218 right: 240 bottom: 73 top: 96 aft: 73 front: 96 rank = 23, NbrRanks are: L: 22, R: 24, B:13, T:33, A:3, F:43!! rank = 23: xs = 180, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 23: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 23: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 11 Cylinder: neighbor = left: 218 right: 120 bottom: 96 top: 117 aft: 73 front: 48 rank = 11, NbrRanks are: L: 10, R: 12, B:1, T:21, A:-9, F:31!! rank = 11: xs = 60, ys = 96, zs = 0, mx = 600, my = 192, mz = 192! rank = 11: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0 rank = 11: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 50 Cylinder: neighbor = left: 218 right: 60 bottom: 96 top: 117 aft: 96 front: 117 rank = 50, NbrRanks are: L: 49, R: 51, B:40, T:60, A:30, F:70!! rank = 50: xs = 0, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 50: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 50: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 51 Cylinder: neighbor = left: 218 right: 120 bottom: 96 top: 117 aft: 96 front: 117 rank = 51, NbrRanks are: L: 50, R: 52, B:41, T:61, A:31, F:71!! rank = 51: xs = 60, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 51: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 51: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 52 Cylinder: neighbor = left: 218 right: 180 bottom: 96 top: 117 aft: 96 front: 117 rank = 52, NbrRanks are: L: 51, R: 53, B:42, T:62, A:32, F:72!! rank = 52: xs = 120, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 52: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 52: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 49 Cylinder: neighbor = left: 540 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank = 49, NbrRanks are: L: 48, R: 50, B:39, T:59, A:29, F:69!! rank = 49: xs = 540, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 49: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 49: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 39 Cylinder: neighbor = left: 540 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank = 39, NbrRanks are: L: 38, R: 40, B:29, T:49, A:19, F:59!! rank = 39: xs = 540, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 39: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 39: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 46 Cylinder: neighbor = left: 360 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank 47 Cylinder: neighbor = left: 420 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank = 47, NbrRanks are: L: 46, R: 48, B:37, T:57, A:27, F:67!! rank = 47: xs = 420, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 47: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 47: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank = 46, NbrRanks are: L: 45, R: 47, B:36, T:56, A:26, F:66!! rank = 46: xs = 360, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 46: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 46: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 74 Cylinder: neighbor = left: 240 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 74, NbrRanks are: L: 73, R: 75, B:64, T:84, A:54, F:94!! rank = 74: xs = 240, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 74: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 74: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 26 Cylinder: neighbor = left: 360 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 26, NbrRanks are: L: 25, R: 27, B:16, T:36, A:6, F:46!! rank = 26: xs = 360, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 26: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 26: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 28 Cylinder: neighbor = left: 480 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 28, NbrRanks are: L: 27, R: 29, B:18, T:38, A:8, F:48!! rank = 28: xs = 480, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 28: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 28: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 27 Cylinder: neighbor = left: 420 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 27, NbrRanks are: L: 26, R: 28, B:17, T:37, A:7, F:47!! rank = 27: xs = 420, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 27: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 27: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 29 Cylinder: neighbor = left: 540 right: 262 bottom: 73 top: 96 aft: 73 front: 96 rank = 29, NbrRanks are: L: 28, R: 30, B:19, T:39, A:9, F:49!! rank = 29: xs = 540, ys = 0, zs = 48, mx = 600, my = 192, mz = 192! rank = 29: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 29: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 66 Cylinder: neighbor = left: 360 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 66, NbrRanks are: L: 65, R: 67, B:56, T:76, A:46, F:86!! rank = 66: xs = 360, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 66: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 66: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 67 Cylinder: neighbor = left: 420 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 67, NbrRanks are: L: 66, R: 68, B:57, T:77, A:47, F:87!! rank = 67: xs = 420, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 67: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 67: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 68 Cylinder: neighbor = left: 480 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 68, NbrRanks are: L: 67, R: 69, B:58, T:78, A:48, F:88!! rank = 68: xs = 480, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 68: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 68: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 69 Cylinder: neighbor = left: 540 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 69, NbrRanks are: L: 68, R: 70, B:59, T:79, A:49, F:89!! rank = 69: xs = 540, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 69: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 69: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 73 Cylinder: neighbor = left: 218 right: 240 bottom: 96 top: 117 aft: 144 front: 117 rank = 73, NbrRanks are: L: 72, R: 74, B:63, T:83, A:53, F:93!! rank = 73: xs = 180, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 73: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 73: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 72 Cylinder: neighbor = left: 218 right: 180 bottom: 96 top: 117 aft: 144 front: 117 rank = 72, NbrRanks are: L: 71, R: 73, B:62, T:82, A:52, F:92!! rank = 72: xs = 120, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 72: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 72: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 37 Cylinder: neighbor = left: 420 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank = 37, NbrRanks are: L: 36, R: 38, B:27, T:47, A:17, F:57!! rank = 37: xs = 420, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 37: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 37: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 55 Cylinder: neighbor = left: 300 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 55, NbrRanks are: L: 54, R: 56, B:45, T:65, A:35, F:75!! rank = 55: xs = 300, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 55: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 55: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 48 Cylinder: neighbor = left: 480 right: 262 bottom: 73 top: 96 aft: 96 front: 117 rank = 48, NbrRanks are: L: 47, R: 49, B:38, T:58, A:28, F:68!! rank = 48: xs = 480, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 48: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 48: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 38 Cylinder: neighbor = left: 480 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank = 38, NbrRanks are: L: 37, R: 39, B:28, T:48, A:18, F:58!! rank = 38: xs = 480, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 38: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 38: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 36 Cylinder: neighbor = left: 360 right: 262 bottom: 96 top: 117 aft: 73 front: 96 rank = 36, NbrRanks are: L: 35, R: 37, B:26, T:46, A:16, F:56!! rank = 36: xs = 360, ys = 96, zs = 48, mx = 600, my = 192, mz = 192! rank = 36: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 36: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 76 Cylinder: neighbor = left: 360 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 76, NbrRanks are: L: 75, R: 77, B:66, T:86, A:56, F:96!! rank = 76: xs = 360, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 76: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 76: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 77 Cylinder: neighbor = left: 420 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 77, NbrRanks are: L: 76, R: 78, B:67, T:87, A:57, F:97!! rank = 77: xs = 420, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 77: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 77: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 78 Cylinder: neighbor = left: 480 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 78, NbrRanks are: L: 77, R: 79, B:68, T:88, A:58, F:98!! rank = 78: xs = 480, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 78: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 78: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 79 Cylinder: neighbor = left: 540 right: 262 bottom: 96 top: 117 aft: 144 front: 117 rank = 79, NbrRanks are: L: 78, R: 80, B:69, T:89, A:59, F:99!! rank = 79: xs = 540, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 79: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 79: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 42 Cylinder: neighbor = left: 218 right: 180 bottom: 73 top: 96 aft: 96 front: 117 rank = 42, NbrRanks are: L: 41, R: 43, B:32, T:52, A:22, F:62!! rank = 42: xs = 120, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 42: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 42: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 43 Cylinder: neighbor = left: 218 right: 240 bottom: 73 top: 96 aft: 96 front: 117 rank = 43, NbrRanks are: L: 42, R: 44, B:33, T:53, A:23, F:63!! rank = 43: xs = 180, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 43: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 43: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 64 Cylinder: neighbor = left: 240 right: 262 bottom: 73 top: 96 aft: 144 front: 117 rank = 64, NbrRanks are: L: 63, R: 65, B:54, T:74, A:44, F:84!! rank = 64: xs = 240, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 64: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 64: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 70 Cylinder: neighbor = left: 218 right: 60 bottom: 96 top: 117 aft: 144 front: 117 rank = 70, NbrRanks are: L: 69, R: 71, B:60, T:80, A:50, F:90!! rank = 70: xs = 0, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 70: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 70: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 71 Cylinder: neighbor = left: 218 right: 120 bottom: 96 top: 117 aft: 144 front: 117 rank = 71, NbrRanks are: L: 70, R: 72, B:61, T:81, A:51, F:91!! rank = 71: xs = 60, ys = 96, zs = 144, mx = 600, my = 192, mz = 192! rank = 71: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1 rank = 71: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 40 Cylinder: neighbor = left: 218 right: 60 bottom: 73 top: 96 aft: 96 front: 117 rank = 40, NbrRanks are: L: 39, R: 41, B:30, T:50, A:20, F:60!! rank = 40: xs = 0, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 40: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 40: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 41 Cylinder: neighbor = left: 218 right: 120 bottom: 73 top: 96 aft: 96 front: 117 rank = 41, NbrRanks are: L: 40, R: 42, B:31, T:51, A:21, F:61!! rank = 41: xs = 60, ys = 0, zs = 96, mx = 600, my = 192, mz = 192! rank = 41: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0 rank = 41: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 58 Cylinder: neighbor = left: 480 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 58, NbrRanks are: L: 57, R: 59, B:48, T:68, A:38, F:78!! rank = 58: xs = 480, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 58: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 58: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 61 Cylinder: neighbor = left: 218 right: 120 bottom: 73 top: 96 aft: 144 front: 117 rank = 61, NbrRanks are: L: 60, R: 62, B:51, T:71, A:41, F:81!! rank = 61: xs = 60, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 61: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 61: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 62 Cylinder: neighbor = left: 218 right: 180 bottom: 73 top: 96 aft: 144 front: 117 rank = 62, NbrRanks are: L: 61, R: 63, B:52, T:72, A:42, F:82!! rank = 62: xs = 120, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 62: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 62: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 60 Cylinder: neighbor = left: 218 right: 60 bottom: 73 top: 96 aft: 144 front: 117 rank = 60, NbrRanks are: L: 59, R: 61, B:50, T:70, A:40, F:80!! rank = 60: xs = 0, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 60: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 60: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 59 Cylinder: neighbor = left: 540 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 59, NbrRanks are: L: 58, R: 60, B:49, T:69, A:39, F:79!! rank = 59: xs = 540, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 59: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 59: xc = 59, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 57 Cylinder: neighbor = left: 420 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 57, NbrRanks are: L: 56, R: 58, B:47, T:67, A:37, F:77!! rank = 57: xs = 420, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 57: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 57: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 63 Cylinder: neighbor = left: 218 right: 240 bottom: 73 top: 96 aft: 144 front: 117 rank = 63, NbrRanks are: L: 62, R: 64, B:53, T:73, A:43, F:83!! rank = 63: xs = 180, ys = 0, zs = 144, mx = 600, my = 192, mz = 192! rank = 63: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1 rank = 63: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 rank 56 Cylinder: neighbor = left: 360 right: 262 bottom: 96 top: 117 aft: 96 front: 117 rank = 56, NbrRanks are: L: 55, R: 57, B:46, T:66, A:36, F:76!! rank = 56: xs = 360, ys = 96, zs = 96, mx = 600, my = 192, mz = 192! rank = 56: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0 rank = 56: xc = 60, yc = 96, zc = 48, xw = 60, yw = 96, zw = 48 Current time step = 20000, Total time step number = 3 0 KSP preconditioned resid norm 7.947822760847e+07 true resid norm 7.036442982861e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 5.470852823762e+06 true resid norm 9.934755238973e+03 ||r(i)||/||b|| 1.411900197752e+00 2 KSP preconditioned resid norm 1.842259488536e+06 true resid norm 6.358986350743e+03 ||r(i)||/||b|| 9.037217193733e-01 3 KSP preconditioned resid norm 2.682886467297e+05 true resid norm 3.363426540473e+03 ||r(i)||/||b|| 4.780009656393e-01 4 KSP preconditioned resid norm 1.600893054855e+05 true resid norm 1.742419112224e+03 ||r(i)||/||b|| 2.476278307758e-01 5 KSP preconditioned resid norm 3.430020787822e+04 true resid norm 8.938161457114e+02 ||r(i)||/||b|| 1.270267019698e-01 6 KSP preconditioned resid norm 2.554860581719e+04 true resid norm 4.639497463673e+02 ||r(i)||/||b|| 6.593526693777e-02 7 KSP preconditioned resid norm 1.075631410435e+04 true resid norm 2.434060854597e+02 ||r(i)||/||b|| 3.459220604111e-02 8 KSP preconditioned resid norm 4.566027667285e+03 true resid norm 1.278963038094e+02 ||r(i)||/||b|| 1.817627231841e-02 9 KSP preconditioned resid norm 2.608692883270e+03 true resid norm 6.657584708911e+01 ||r(i)||/||b|| 9.461577000093e-03 10 KSP preconditioned resid norm 1.000117603355e+03 true resid norm 3.482621094833e+01 ||r(i)||/||b|| 4.949405691648e-03 11 KSP preconditioned resid norm 5.586892411083e+02 true resid norm 1.833456557997e+01 ||r(i)||/||b|| 2.605658231670e-03 12 KSP preconditioned resid norm 3.151979364930e+02 true resid norm 9.682933287542e+00 ||r(i)||/||b|| 1.376111951895e-03 13 KSP preconditioned resid norm 1.450159326994e+02 true resid norm 5.109117861524e+00 ||r(i)||/||b|| 7.260938337693e-04 14 KSP preconditioned resid norm 4.700842649561e+01 true resid norm 2.675610170027e+00 ||r(i)||/||b|| 3.802503873824e-04 15 KSP preconditioned resid norm 2.876898213652e+01 true resid norm 1.400139965989e+00 ||r(i)||/||b|| 1.989840562056e-04 16 KSP preconditioned resid norm 3.348130992519e+01 true resid norm 7.362133421925e-01 ||r(i)||/||b|| 1.046286232953e-04 17 KSP preconditioned resid norm 1.816532639830e+01 true resid norm 3.874617890624e-01 ||r(i)||/||b|| 5.506500798858e-05 18 KSP preconditioned resid norm 5.648794390890e+00 true resid norm 2.041017634279e-01 ||r(i)||/||b|| 2.900638347032e-05 Linear solve converged due to CONVERGED_RTOL iterations 18 Residual norm 0.204102 0 KSP preconditioned resid norm 3.856433808270e+08 true resid norm 5.716853866429e+03 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.857890665990e+06 true resid norm 3.549680244437e+04 ||r(i)||/||b|| 6.209149870494e+00 2 KSP preconditioned resid norm 2.891960128406e+06 true resid norm 2.266072012757e+04 ||r(i)||/||b|| 3.963844565040e+00 3 KSP preconditioned resid norm 1.112741883891e+06 true resid norm 1.207987951783e+04 ||r(i)||/||b|| 2.113029264010e+00 4 KSP preconditioned resid norm 3.119525682570e+05 true resid norm 6.306183467702e+03 ||r(i)||/||b|| 1.103086350472e+00 5 KSP preconditioned resid norm 1.444352684110e+05 true resid norm 3.265520796802e+03 ||r(i)||/||b|| 5.712094227174e-01 6 KSP preconditioned resid norm 8.885428679430e+04 true resid norm 1.691812755024e+03 ||r(i)||/||b|| 2.959342314064e-01 7 KSP preconditioned resid norm 4.807698986323e+04 true resid norm 8.850486020031e+02 ||r(i)||/||b|| 1.548139278494e-01 8 KSP preconditioned resid norm 2.351085988243e+04 true resid norm 4.656909285552e+02 ||r(i)||/||b|| 8.145930251775e-02 9 KSP preconditioned resid norm 8.796933340228e+03 true resid norm 2.437636722654e+02 ||r(i)||/||b|| 4.263947932916e-02 10 KSP preconditioned resid norm 2.834224931076e+03 true resid norm 1.282352374177e+02 ||r(i)||/||b|| 2.243108542107e-02 11 KSP preconditioned resid norm 2.063748127973e+03 true resid norm 6.788719044571e+01 ||r(i)||/||b|| 1.187492142214e-02 12 KSP preconditioned resid norm 1.353971718729e+03 true resid norm 3.613942954257e+01 ||r(i)||/||b|| 6.321559092981e-03 13 KSP preconditioned resid norm 5.835542424980e+02 true resid norm 1.921943033918e+01 ||r(i)||/||b|| 3.361889386755e-03 14 KSP preconditioned resid norm 1.932255447921e+02 true resid norm 1.022994296024e+01 ||r(i)||/||b|| 1.789435797951e-03 15 KSP preconditioned resid norm 9.062967041491e+01 true resid norm 5.422901897037e+00 ||r(i)||/||b|| 9.485815141930e-04 16 KSP preconditioned resid norm 6.085778763605e+01 true resid norm 2.863857196164e+00 ||r(i)||/||b|| 5.009498691197e-04 17 KSP preconditioned resid norm 3.660177659870e+01 true resid norm 1.502741027406e+00 ||r(i)||/||b|| 2.628615428199e-04 Linear solve converged due to CONVERGED_RTOL iterations 17 Residual norm 1.50274 Rank = 48 is ready to communicate!! Rank = 64 is ready to communicate!! Rank = 0 is ready to communicate!! Rank = 24 is ready to communicate!! Rank = 32 is ready to communicate!! Rank = 49 is ready to communicate!! Rank = 72 is ready to communicate!! Rank = 65 is ready to communicate!! Rank = 1 is ready to communicate!! Rank = 16 is ready to communicate!! Rank = 8 is ready to communicate!! Rank = 25 is ready to communicate!! Rank = 33 is ready to communicate!! Rank = 50 is ready to communicate!! Rank = 74 is ready to communicate!! Rank = 66 is ready to communicate!! Rank = 40 is ready to communicate!! Rank = 2 is ready to communicate!! Rank = 56 is ready to communicate!! Rank = 17 is ready to communicate!! Rank = 9 is ready to communicate!! Rank = 26 is ready to communicate!! Rank = 36 is ready to communicate!! Rank = 52 is ready to communicate!! Rank = 76 is ready to communicate!! Rank = 68 is ready to communicate!! Rank = 41 is ready to communicate!! Rank = 4 is ready to communicate!! Rank = 60 is ready to communicate!! Rank = 18 is ready to communicate!! Rank = 10 is ready to communicate!! Rank = 28 is ready to communicate!! Rank = 34 is ready to communicate!! Rank = 51 is ready to communicate!! Rank = 75 is ready to communicate!! Rank = 67 is ready to communicate!! Rank = 44 is ready to communicate!! Rank = 3 is ready to communicate!! Rank = 57 is ready to communicate!! Rank = 20 is ready to communicate!! Rank = 12 is ready to communicate!! Rank = 27 is ready to communicate!! Rank = 35 is ready to communicate!! Rank = 54 is ready to communicate!! Rank = 78 is ready to communicate!! Rank = 70 is ready to communicate!! Rank = 42 is ready to communicate!! Rank = 5 is ready to communicate!! Rank = 58 is ready to communicate!! Rank = 19 is ready to communicate!! Rank = 11 is ready to communicate!! Rank = 30 is ready to communicate!! Rank = 38 is ready to communicate!! Rank = 53 is ready to communicate!! Rank = 73 is ready to communicate!! Rank = 69 is ready to communicate!! Rank = 45 is ready to communicate!! Rank = 6 is ready to communicate!! Rank = 62 is ready to communicate!! Rank = 22 is ready to communicate!! Rank = 13 is ready to communicate!! Rank = 31 is ready to communicate!! Rank = 37 is ready to communicate!! Rank = 55 is ready to communicate!! Rank = 77 is ready to communicate!! Rank = 71 is ready to communicate!! Rank = 46 is ready to communicate!! Rank = 7 is ready to communicate!! Rank = 61 is ready to communicate!! Rank = 21 is ready to communicate!! Rank = 14 is ready to communicate!! Rank = 29 is ready to communicate!! Rank = 39 is ready to communicate!! Rank = 79 is ready to communicate!! Rank = 43 is ready to communicate!! Rank = 63 is ready to communicate!! Rank = 23 is ready to communicate!! Rank = 15 is ready to communicate!! Rank = 47 is ready to communicate!! Rank = 59 is ready to communicate!! PlotU is ready!! PlotU is finished!! Current time step = 20001, Total time step number = 3 0 KSP preconditioned resid norm 5.155842822045e+07 true resid norm 9.333808441594e+04 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.122574641651e+07 true resid norm 3.531829738596e+04 ||r(i)||/||b|| 3.783910673437e-01 2 KSP preconditioned resid norm 7.137871997817e+06 true resid norm 1.684356866013e+04 ||r(i)||/||b|| 1.804576209757e-01 3 KSP preconditioned resid norm 2.407636573843e+06 true resid norm 8.436938669292e+03 ||r(i)||/||b|| 9.039117014331e-02 4 KSP preconditioned resid norm 9.005148376957e+05 true resid norm 4.311315128868e+03 ||r(i)||/||b|| 4.619031080235e-02 5 KSP preconditioned resid norm 3.394158972731e+05 true resid norm 2.245652159484e+03 ||r(i)||/||b|| 2.405933412428e-02 6 KSP preconditioned resid norm 1.070882656514e+05 true resid norm 1.188543952702e+03 ||r(i)||/||b|| 1.273375128855e-02 7 KSP preconditioned resid norm 2.234484381934e+04 true resid norm 6.332789636681e+02 ||r(i)||/||b|| 6.784786377723e-03 8 KSP preconditioned resid norm 2.034115899955e+04 true resid norm 3.361030627867e+02 ||r(i)||/||b|| 3.600920941220e-03 9 KSP preconditioned resid norm 1.929849669956e+04 true resid norm 1.772561512031e+02 ||r(i)||/||b|| 1.899076377154e-03 10 KSP preconditioned resid norm 8.724455049462e+03 true resid norm 9.364204984144e+01 ||r(i)||/||b|| 1.003256606640e-03 11 KSP preconditioned resid norm 2.939406179032e+03 true resid norm 4.936630261030e+01 ||r(i)||/||b|| 5.288977475722e-04 12 KSP preconditioned resid norm 5.941865008084e+02 true resid norm 2.596413495447e+01 ||r(i)||/||b|| 2.781729999811e-04 13 KSP preconditioned resid norm 4.514769456075e+02 true resid norm 1.369529930468e+01 ||r(i)||/||b|| 1.467278805900e-04 14 KSP preconditioned resid norm 3.507616600328e+02 true resid norm 7.263138664258e+00 ||r(i)||/||b|| 7.781538168162e-05 15 KSP preconditioned resid norm 1.816967941005e+02 true resid norm 3.839946314104e+00 ||r(i)||/||b|| 4.114018771793e-05 16 KSP preconditioned resid norm 5.219692565566e+01 true resid norm 2.023635100009e+00 ||r(i)||/||b|| 2.168070099865e-05 17 KSP preconditioned resid norm 1.800506392895e+01 true resid norm 1.073035257174e+00 ||r(i)||/||b|| 1.149622111797e-05 18 KSP preconditioned resid norm 1.608501987990e+01 true resid norm 5.692446152237e-01 ||r(i)||/||b|| 6.098738995831e-06 19 KSP preconditioned resid norm 7.553482181553e+00 true resid norm 3.013653959652e-01 ||r(i)||/||b|| 3.228750598976e-06 20 KSP preconditioned resid norm 2.264934352257e+00 true resid norm 1.593648680106e-01 ||r(i)||/||b|| 1.707393814731e-06 Linear solve converged due to CONVERGED_RTOL iterations 20 Residual norm 0.159365 0 KSP preconditioned resid norm 2.473326511036e+09 true resid norm 5.504707892312e+04 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 4.395030304173e+07 true resid norm 2.529103873168e+05 ||r(i)||/||b|| 4.594437929577e+00 2 KSP preconditioned resid norm 2.201564602935e+07 true resid norm 1.558635504651e+05 ||r(i)||/||b|| 2.831459062210e+00 3 KSP preconditioned resid norm 1.057684829931e+07 true resid norm 8.265169245697e+04 ||r(i)||/||b|| 1.501472813342e+00 4 KSP preconditioned resid norm 2.567381455158e+06 true resid norm 4.349931862509e+04 ||r(i)||/||b|| 7.902202891790e-01 5 KSP preconditioned resid norm 8.920244129377e+05 true resid norm 2.275702748540e+04 ||r(i)||/||b|| 4.134102650058e-01 6 KSP preconditioned resid norm 5.340725623770e+05 true resid norm 1.176413341027e+04 ||r(i)||/||b|| 2.137104028117e-01 7 KSP preconditioned resid norm 3.080112173933e+05 true resid norm 6.123203452052e+03 ||r(i)||/||b|| 1.112357562261e-01 8 KSP preconditioned resid norm 1.863632964060e+05 true resid norm 3.224285512234e+03 ||r(i)||/||b|| 5.857323540703e-02 9 KSP preconditioned resid norm 7.315697630444e+04 true resid norm 1.694365339391e+03 ||r(i)||/||b|| 3.078029520436e-02 10 KSP preconditioned resid norm 2.283404118378e+04 true resid norm 8.934818280717e+02 ||r(i)||/||b|| 1.623123053123e-02 11 KSP preconditioned resid norm 1.147133770622e+04 true resid norm 4.733002173332e+02 ||r(i)||/||b|| 8.598098692833e-03 12 KSP preconditioned resid norm 8.699788257074e+03 true resid norm 2.522786104650e+02 ||r(i)||/||b|| 4.582960901838e-03 13 KSP preconditioned resid norm 4.192702781806e+03 true resid norm 1.345447634140e+02 ||r(i)||/||b|| 2.444176258688e-03 14 KSP preconditioned resid norm 1.471147423099e+03 true resid norm 7.189377117936e+01 ||r(i)||/||b|| 1.306041530010e-03 15 KSP preconditioned resid norm 6.000761108344e+02 true resid norm 3.821031722333e+01 ||r(i)||/||b|| 6.941388711414e-04 16 KSP preconditioned resid norm 3.625405849885e+02 true resid norm 2.020652141929e+01 ||r(i)||/||b|| 3.670770877326e-04 17 KSP preconditioned resid norm 2.331718436433e+02 true resid norm 1.060307066842e+01 ||r(i)||/||b|| 1.926182256324e-04 Linear solve converged due to CONVERGED_RTOL iterations 17 Residual norm 10.6031 Rank = 16 is ready to communicate!! Rank = 0 is ready to communicate!! Rank = 17 is ready to communicate!! Rank = 24 is ready to communicate!! Rank = 48 is ready to communicate!! Rank = 64 is ready to communicate!! Rank = 1 is ready to communicate!! Rank = 18 is ready to communicate!! Rank = 8 is ready to communicate!! Rank = 25 is ready to communicate!! Rank = 32 is ready to communicate!! Rank = 50 is ready to communicate!! Rank = 72 is ready to communicate!! Rank = 65 is ready to communicate!! Rank = 2 is ready to communicate!! Rank = 20 is ready to communicate!! Rank = 9 is ready to communicate!! Rank = 26 is ready to communicate!! Rank = 33 is ready to communicate!! Rank = 51 is ready to communicate!! Rank = 73 is ready to communicate!! Rank = 66 is ready to communicate!! Rank = 3 is ready to communicate!! Rank = 22 is ready to communicate!! Rank = 12 is ready to communicate!! Rank = 28 is ready to communicate!! Rank = 34 is ready to communicate!! Rank = 52 is ready to communicate!! Rank = 74 is ready to communicate!! Rank = 67 is ready to communicate!! Rank = 4 is ready to communicate!! Rank = 19 is ready to communicate!! Rank = 13 is ready to communicate!! Rank = 27 is ready to communicate!! Rank = 35 is ready to communicate!! Rank = 53 is ready to communicate!! Rank = 75 is ready to communicate!! Rank = 68 is ready to communicate!! Rank = 5 is ready to communicate!! Rank = 56 is ready to communicate!! Rank = 21 is ready to communicate!! Rank = 10 is ready to communicate!! Rank = 29 is ready to communicate!! Rank = 36 is ready to communicate!! Rank = 49 is ready to communicate!! Rank = 76 is ready to communicate!! Rank = 69 is ready to communicate!! Rank = 40 is ready to communicate!! Rank = 6 is ready to communicate!! Rank = 23 is ready to communicate!! Rank = 11 is ready to communicate!! Rank = 30 is ready to communicate!! Rank = 37 is ready to communicate!! Rank = 54 is ready to communicate!! Rank = 78 is ready to communicate!! Rank = 70 is ready to communicate!! Rank = 41 is ready to communicate!! Rank = 7 is ready to communicate!! Rank = 14 is ready to communicate!! Rank = 31 is ready to communicate!! Rank = 38 is ready to communicate!! Rank = 55 is ready to communicate!! Rank = 77 is ready to communicate!! Rank = 71 is ready to communicate!! Rank = 42 is ready to communicate!! Rank = 15 is ready to communicate!! Rank = 39 is ready to communicate!! Rank = 79 is ready to communicate!! Rank = 43 is ready to communicate!! Rank = 57 is ready to communicate!! Rank = 44 is ready to communicate!! Rank = 60 is ready to communicate!! Rank = 45 is ready to communicate!! Rank = 61 is ready to communicate!! Rank = 46 is ready to communicate!! Rank = 58 is ready to communicate!! Rank = 47 is ready to communicate!! Rank = 59 is ready to communicate!! Rank = 62 is ready to communicate!! Rank = 63 is ready to communicate!! PlotU is ready!! PlotU is finished!! Current time step = 20002, Total time step number = 3 0 KSP preconditioned resid norm 5.250869240111e+09 true resid norm 5.419979971295e+07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 2.656445473557e+09 true resid norm 7.927739716578e+06 ||r(i)||/||b|| 1.462688009654e-01 2 KSP preconditioned resid norm 1.140646069475e+09 true resid norm 3.152722813837e+06 ||r(i)||/||b|| 5.816853255057e-02 3 KSP preconditioned resid norm 3.623951092624e+08 true resid norm 1.565180472377e+06 ||r(i)||/||b|| 2.887797520777e-02 4 KSP preconditioned resid norm 9.275414350155e+07 true resid norm 7.897979251599e+05 ||r(i)||/||b|| 1.457197128666e-02 5 KSP preconditioned resid norm 2.879198321255e+07 true resid norm 4.067779499670e+05 ||r(i)||/||b|| 7.505155962224e-03 6 KSP preconditioned resid norm 9.048310261211e+06 true resid norm 2.107051864111e+05 ||r(i)||/||b|| 3.887563930623e-03 7 KSP preconditioned resid norm 1.841646992836e+06 true resid norm 1.095410507080e+05 ||r(i)||/||b|| 2.021060064579e-03 8 KSP preconditioned resid norm 1.844200422319e+06 true resid norm 5.683536085013e+04 ||r(i)||/||b|| 1.048626768939e-03 9 KSP preconditioned resid norm 1.828472592777e+06 true resid norm 2.916494861380e+04 ||r(i)||/||b|| 5.381006713726e-04 10 KSP preconditioned resid norm 9.711083417622e+05 true resid norm 1.509575173929e+04 ||r(i)||/||b|| 2.785204340098e-04 11 KSP preconditioned resid norm 3.172311945097e+05 true resid norm 7.819131085911e+03 ||r(i)||/||b|| 1.442649442862e-04 12 KSP preconditioned resid norm 6.444522937803e+04 true resid norm 4.104332643191e+03 ||r(i)||/||b|| 7.572597435652e-05 13 KSP preconditioned resid norm 3.676943658439e+04 true resid norm 2.170745900883e+03 ||r(i)||/||b|| 4.005081037900e-05 14 KSP preconditioned resid norm 3.019960385044e+04 true resid norm 1.151642750654e+03 ||r(i)||/||b|| 2.124809974858e-05 15 KSP preconditioned resid norm 1.676685243047e+04 true resid norm 6.121824889902e+02 ||r(i)||/||b|| 1.129492160917e-05 16 KSP preconditioned resid norm 6.072064987615e+03 true resid norm 3.231288682737e+02 ||r(i)||/||b|| 5.961809268393e-06 17 KSP preconditioned resid norm 1.774235054927e+03 true resid norm 1.712579655020e+02 ||r(i)||/||b|| 3.159752737261e-06 18 KSP preconditioned resid norm 1.585174072892e+03 true resid norm 9.082516976691e+01 ||r(i)||/||b|| 1.675747332056e-06 19 KSP preconditioned resid norm 7.486808666010e+02 true resid norm 4.805558999260e+01 ||r(i)||/||b|| 8.866377781304e-07 20 KSP preconditioned resid norm 2.450041463835e+02 true resid norm 2.539192900000e+01 ||r(i)||/||b|| 4.684875061252e-07 Linear solve converged due to CONVERGED_RTOL iterations 20 Residual norm 25.3919 0 KSP preconditioned resid norm 1.308343931964e+11 true resid norm 3.379884097346e+07 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 3.147149592544e+10 true resid norm 1.911788601668e+07 ||r(i)||/||b|| 5.656373256021e-01 2 KSP preconditioned resid norm 3.300712243956e+09 true resid norm 1.011185466473e+07 ||r(i)||/||b|| 2.991775567887e-01 3 KSP preconditioned resid norm 6.574698764046e+08 true resid norm 5.301213909160e+06 ||r(i)||/||b|| 1.568460265641e-01 4 KSP preconditioned resid norm 4.628784604268e+08 true resid norm 2.799453142464e+06 ||r(i)||/||b|| 8.282689766380e-02 5 KSP preconditioned resid norm 8.948293494213e+07 true resid norm 1.463886466697e+06 ||r(i)||/||b|| 4.331173568485e-02 6 KSP preconditioned resid norm 2.633080756623e+07 true resid norm 7.602042358184e+05 ||r(i)||/||b|| 2.249202084815e-02 7 KSP preconditioned resid norm 1.524610814922e+07 true resid norm 3.947570010698e+05 ||r(i)||/||b|| 1.167960171711e-02 8 KSP preconditioned resid norm 8.531401658692e+06 true resid norm 2.074658215241e+05 ||r(i)||/||b|| 6.138252542062e-03 9 KSP preconditioned resid norm 3.978994257051e+06 true resid norm 1.088783408169e+05 ||r(i)||/||b|| 3.221363149772e-03 10 KSP preconditioned resid norm 2.008808476507e+06 true resid norm 5.739289970933e+04 ||r(i)||/||b|| 1.698073012456e-03 11 KSP preconditioned resid norm 6.843646614866e+05 true resid norm 3.044560507849e+04 ||r(i)||/||b|| 9.007884353904e-04 12 KSP preconditioned resid norm 2.904179583089e+05 true resid norm 1.624861072438e+04 ||r(i)||/||b|| 4.807446130220e-04 13 KSP preconditioned resid norm 1.636960447152e+05 true resid norm 8.676038844253e+03 ||r(i)||/||b|| 2.566963420747e-04 14 KSP preconditioned resid norm 7.687345087387e+04 true resid norm 4.621718459149e+03 ||r(i)||/||b|| 1.367419214990e-04 15 KSP preconditioned resid norm 4.412626033081e+04 true resid norm 2.447040027843e+03 ||r(i)||/||b|| 7.240011661242e-05 16 KSP preconditioned resid norm 1.999982111233e+04 true resid norm 1.292038360004e+03 ||r(i)||/||b|| 3.822729782415e-05 17 KSP preconditioned resid norm 1.225879258713e+04 true resid norm 6.805313286722e+02 ||r(i)||/||b|| 2.013475341378e-05 Linear solve converged due to CONVERGED_RTOL iterations 17 Residual norm 680.531 Rank = 48 is ready to communicate!! Rank = 0 is ready to communicate!! Rank = 8 is ready to communicate!! Rank = 24 is ready to communicate!! Rank = 32 is ready to communicate!! Rank = 49 is ready to communicate!! Rank = 72 is ready to communicate!! Rank = 64 is ready to communicate!! Rank = 1 is ready to communicate!! Rank = 16 is ready to communicate!! Rank = 9 is ready to communicate!! Rank = 26 is ready to communicate!! Rank = 34 is ready to communicate!! Rank = 52 is ready to communicate!! Rank = 73 is ready to communicate!! Rank = 65 is ready to communicate!! Rank = 40 is ready to communicate!! Rank = 2 is ready to communicate!! Rank = 20 is ready to communicate!! Rank = 12 is ready to communicate!! Rank = 28 is ready to communicate!! Rank = 33 is ready to communicate!! Rank = 50 is ready to communicate!! Rank = 76 is ready to communicate!! Rank = 68 is ready to communicate!! Rank = 41 is ready to communicate!! Rank = 4 is ready to communicate!! Rank = 56 is ready to communicate!! Rank = 18 is ready to communicate!! Rank = 13 is ready to communicate!! Rank = 27 is ready to communicate!! Rank = 36 is ready to communicate!! Rank = 53 is ready to communicate!! Rank = 74 is ready to communicate!! Rank = 66 is ready to communicate!! Rank = 44 is ready to communicate!! Rank = 3 is ready to communicate!! Rank = 57 is ready to communicate!! Rank = 19 is ready to communicate!! Rank = 14 is ready to communicate!! Rank = 25 is ready to communicate!! Rank = 35 is ready to communicate!! Rank = 54 is ready to communicate!! Rank = 77 is ready to communicate!! Rank = 67 is ready to communicate!! Rank = 45 is ready to communicate!! Rank = 5 is ready to communicate!! Rank = 58 is ready to communicate!! Rank = 17 is ready to communicate!! Rank = 10 is ready to communicate!! Rank = 29 is ready to communicate!! Rank = 37 is ready to communicate!! Rank = 51 is ready to communicate!! Rank = 75 is ready to communicate!! Rank = 69 is ready to communicate!! Rank = 42 is ready to communicate!! Rank = 6 is ready to communicate!! Rank = 60 is ready to communicate!! Rank = 22 is ready to communicate!! Rank = 11 is ready to communicate!! Rank = 30 is ready to communicate!! Rank = 38 is ready to communicate!! Rank = 55 is ready to communicate!! Rank = 78 is ready to communicate!! Rank = 70 is ready to communicate!! Rank = 46 is ready to communicate!! Rank = 7 is ready to communicate!! Rank = 61 is ready to communicate!! Rank = 23 is ready to communicate!! Rank = 15 is ready to communicate!! Rank = 31 is ready to communicate!! Rank = 39 is ready to communicate!! Rank = 79 is ready to communicate!! Rank = 71 is ready to communicate!! Rank = 43 is ready to communicate!! Rank = 62 is ready to communicate!! Rank = 21 is ready to communicate!! Rank = 47 is ready to communicate!! Rank = 63 is ready to communicate!! Rank = 59 is ready to communicate!! PlotU is ready!! PlotU is finished!! ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex45 on a arch-linux2-c-debug named compute-7-13.local with 80 processors, by zlwei Sun Oct 21 20:00:55 2012 Using Petsc Development HG revision: 98bf11863c3be31b7c2af504314a500bc64d88c9 HG Date: Wed Aug 29 13:51:08 2012 -0500 Max Max/Min Avg Total Time (sec): 3.775e+03 1.00001 3.775e+03 Objects: 2.559e+03 1.00000 2.559e+03 Flops: 8.895e+09 1.04223 8.705e+09 6.964e+11 Flops/sec: 2.356e+06 1.04224 2.306e+06 1.845e+08 MPI Messages: 3.805e+04 2.27909 2.994e+04 2.395e+06 MPI Message Lengths: 5.262e+08 1.79828 1.471e+04 3.523e+10 MPI Reductions: 6.043e+03 1.00000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total counts %Total Avg %Total counts %Total 0: Main Stage: 1.8489e+03 49.0% 2.2497e+09 0.3% 1.136e+04 0.5% 4.988e+03 33.9% 2.000e+01 0.3% 1: DMMG Setup: 3.8987e+00 0.1% 0.0000e+00 0.0% 2.768e+03 0.1% 5.355e+00 0.0% 3.500e+01 0.6% 2: Pressure RHS Setup: 5.6124e+02 14.9% 1.6577e+11 23.8% 4.129e+05 17.2% 2.339e+03 15.9% 2.322e+03 38.4% 3: Pressure Solve: 3.7698e+02 10.0% 1.9245e+11 27.6% 8.250e+05 34.4% 2.672e+03 18.2% 7.100e+02 11.7% 4: Corrector RHS Setup: 6.5075e+02 17.2% 1.6577e+11 23.8% 4.129e+05 17.2% 2.339e+03 15.9% 2.322e+03 38.4% 5: Corrector Solve: 3.3330e+02 8.8% 1.7019e+11 24.4% 7.302e+05 30.5% 2.363e+03 16.1% 6.330e+02 10.5% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flops: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent Avg. len: average message length Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %f - percent flops in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flops --- Global --- --- Stage --- Total Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage VecNorm 6 1.0 4.3274e+0010.4 3.32e+06 1.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 12 0 0 30 61 VecSet 1 1.0 9.2571e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 6 1.0 1.5605e-02 3.3 3.32e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 12 0 0 0 17008 VecScatterBegin 7 1.0 1.8286e-02 2.9 0.00e+00 0.0 3.1e+03 3.3e+04 0.0e+00 0 0 0 0 0 0 0 27 1 0 0 VecScatterEnd 7 1.0 5.9846e+001399.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 6 1.0 3.8553e+0047.3 2.15e+07 1.0 2.1e+03 3.7e+04 0.0e+00 0 0 0 0 0 0 76 18 1 0 446 --- Event Stage 1: DMMG Setup --- Event Stage 2: Pressure RHS Setup KSPGMRESOrthog 90 1.0 3.2382e+01 7.7 2.01e+08 1.0 0.0e+00 0.0e+00 9.0e+01 1 2 0 0 1 4 10 0 0 4 493 KSPSetUp 24 1.0 5.4547e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+01 0 0 0 0 1 0 0 0 0 2 0 VecMDot 90 1.0 3.2059e+01 8.1 1.00e+08 1.0 0.0e+00 0.0e+00 9.0e+01 1 1 0 0 1 4 5 0 0 4 249 VecNorm 99 1.0 1.4623e+00 2.4 2.01e+07 1.0 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 2 0 1 0 0 4 1091 VecScale 99 1.0 5.4037e-02 2.1 1.00e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14757 VecCopy 9 1.0 5.0828e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 159 1.0 4.5193e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 9 1.0 9.0234e-03 2.5 1.83e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16067 VecMAXPY 99 1.0 4.5966e-01 1.7 1.19e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 6 0 0 0 20502 VecAssemblyBegin 123 1.0 4.8207e+0118.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+02 1 0 0 0 6 6 0 0 0 16 0 VecAssemblyEnd 123 1.0 3.6597e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 99 1.0 2.0102e-01 3.3 1.00e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3967 VecScatterBegin 228 1.0 1.1732e-01 3.3 0.00e+00 0.0 2.0e+05 1.1e+04 0.0e+00 0 0 8 6 0 0 0 48 38 0 0 VecScatterEnd 228 1.0 1.5576e+02127.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 9 0 0 0 0 0 VecSetRandom 9 1.0 3.7712e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 99 1.0 1.5003e+00 2.3 3.01e+07 1.0 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 2 0 1 0 0 4 1595 MatMult 90 1.0 2.9057e+0143.7 1.62e+08 1.0 6.9e+04 8.0e+03 0.0e+00 0 2 3 2 0 2 8 17 10 0 435 MatConvert 9 1.0 1.8912e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 27 1.0 3.1827e+0021.6 2.37e+07 1.0 6.9e+03 8.0e+03 0.0e+00 0 0 0 0 0 0 1 2 1 0 583 MatAssemblyBegin 153 1.0 1.7952e+02 4.9 0.00e+00 0.0 1.4e+04 2.0e+03 1.7e+02 4 0 1 0 3 24 0 3 0 7 0 MatAssemblyEnd 153 1.0 1.5646e+01 1.9 0.00e+00 0.0 8.3e+04 1.4e+03 4.3e+02 0 0 3 0 7 2 0 20 2 19 0 MatGetRow 1825272 1.0 8.2963e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 9 1.0 8.0746e+01 1.0 0.00e+00 0.0 6.2e+04 1.2e+04 2.7e+02 2 0 3 2 4 14 0 15 14 12 0 MatAXPY 9 1.0 7.1768e-02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 9 1.0 1.1946e+01 1.0 1.71e+07 1.0 4.4e+04 3.9e+03 2.2e+02 0 0 2 0 4 2 1 11 3 9 112 MatMatMultSym 9 1.0 8.4525e+00 1.1 0.00e+00 0.0 3.7e+04 3.2e+03 2.0e+02 0 0 2 0 3 1 0 9 2 9 0 MatMatMultNum 9 1.0 3.8606e+00 1.1 1.71e+07 1.0 6.9e+03 8.0e+03 1.8e+01 0 0 0 0 0 1 1 2 1 1 347 MatPtAP 9 1.0 4.8058e+01 1.0 3.24e+08 1.1 7.7e+04 8.0e+03 2.4e+02 1 4 3 2 4 9 15 19 11 10 518 MatPtAPSymbolic 9 1.0 3.7139e+01 1.1 0.00e+00 0.0 6.9e+04 6.8e+03 2.2e+02 1 0 3 1 4 6 0 17 8 10 0 MatPtAPNumeric 9 1.0 1.4548e+01 1.3 3.24e+08 1.1 7.9e+03 1.9e+04 1.8e+01 0 4 0 0 0 2 15 2 3 1 1710 MatTrnMatMult 9 1.0 3.0112e+02 1.0 1.35e+09 1.1 4.5e+04 5.7e+04 2.6e+02 8 15 2 7 4 54 63 11 46 11 346 MatGetLocalMat 45 1.0 5.6984e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 5.4e+01 0 0 0 0 1 0 0 0 0 2 0 MatGetBrAoCol 27 1.0 3.0325e+0125.0 0.00e+00 0.0 4.9e+04 1.1e+04 3.6e+01 0 0 2 2 1 3 0 12 10 2 0 MatGetSymTrans 18 1.0 5.0844e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 3 1.0 5.5368e+02 1.0 2.13e+09 1.1 4.0e+05 1.4e+04 2.1e+03 15 24 17 16 35 99100 97 99 91 299 PCGAMGgraph_AGG 3 1.0 3.2943e+00 1.2 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 1 0 2 0 2 135 PCGAMGcoarse_AGG 3 1.0 1.3759e+02 1.0 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 4 5 2 4 4 24 21 12 23 10 253 PCGAMGProl_AGG 3 1.0 1.2205e+01 1.2 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 2 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.6095e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 3 7 9 4 7 730 PCGAMGgraph_AGG 3 1.0 3.1328e+00 1.2 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 0 0 2 0 2 142 PCGAMGcoarse_AGG 3 1.0 1.5060e+02 1.0 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 4 5 2 4 4 26 21 12 23 10 231 PCGAMGProl_AGG 3 1.0 1.0595e+01 1.8 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 2 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.5662e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 3 7 9 4 7 750 PCGAMGgraph_AGG 3 1.0 3.0721e+00 1.3 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 0 0 2 0 2 145 PCGAMGcoarse_AGG 3 1.0 1.3575e+02 1.0 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 4 5 2 4 4 24 21 12 23 10 256 PCGAMGProl_AGG 3 1.0 1.0627e+01 1.3 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 2 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.5674e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 3 7 9 4 7 749 --- Event Stage 3: Pressure Solve KSPGMRESOrthog 61 1.0 1.6081e-01 2.7 1.06e+05 0.0 0.0e+00 0.0e+00 6.1e+01 0 0 0 0 1 0 0 0 0 9 1 KSPSetUp 3 1.0 3.3379e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 3 1.0 3.7704e+02 1.0 2.44e+09 1.0 8.3e+05 7.8e+03 7.1e+02 10 28 34 18 12 100100100100100 510 VecMDot 61 1.0 1.6046e-01 2.7 5.28e+04 0.0 0.0e+00 0.0e+00 6.1e+01 0 0 0 0 1 0 0 0 0 9 0 VecTDot 116 1.0 4.2943e+0110.4 6.41e+07 1.0 0.0e+00 0.0e+00 1.2e+02 1 1 0 0 2 9 3 0 0 16 119 VecNorm 247 1.0 1.4691e+0216.7 6.92e+07 1.0 0.0e+00 0.0e+00 2.5e+02 2 1 0 0 4 24 3 0 0 35 38 VecScale 122 1.0 1.8764e-04 4.1 5.28e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 282 VecCopy 311 1.0 3.0070e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 863 1.0 1.4365e-01 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 543 1.0 8.4607e-01 2.6 1.38e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 6 0 0 0 13034 VecAYPX 848 1.0 8.2516e-01 2.0 1.22e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 5 0 0 0 11729 VecMAXPY 122 1.0 2.5153e-04 6.0 1.06e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 420 VecAssemblyBegin 3 1.0 2.8673e-01 9.8 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 1 0 VecAssemblyEnd 3 1.0 2.5034e-05 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 366 1.0 5.9826e-01 2.5 3.71e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 4928 VecScatterBegin 1278 1.0 3.8352e-01 2.9 0.00e+00 0.0 8.3e+05 7.8e+03 0.0e+00 0 0 34 18 0 0 0100100 0 0 VecScatterEnd 1278 1.0 3.5239e+02 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 64 0 0 0 0 0 VecNormalize 122 1.0 7.7030e+00136.7 1.58e+05 0.0 0.0e+00 0.0e+00 1.2e+02 0 0 0 0 2 1 0 0 0 17 0 MatMult 912 1.0 3.3210e+02 4.9 1.74e+09 1.0 6.1e+05 9.9e+03 0.0e+00 5 20 25 17 0 47 71 73 94 0 413 MatMultAdd 183 1.0 3.0270e+01 4.1 1.34e+08 1.0 1.1e+05 1.7e+03 0.0e+00 0 2 5 1 0 5 5 13 3 0 348 MatMultTranspose 183 1.0 9.2306e+01 5.5 1.34e+08 1.0 1.1e+05 1.7e+03 0.0e+00 1 2 5 1 0 15 5 13 3 0 114 MatSolve 122 0.0 3.1532e-02 0.0 2.15e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 681 MatLUFactorSym 3 1.0 1.0194e-02199.8 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 1 0 MatLUFactorNum 3 1.0 5.5575e-0211655.0 3.25e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 585 MatGetRowIJ 3 0.0 8.2707e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 3 0.0 2.2392e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.5e-02 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 3 1.0 6.8424e-02133.4 3.25e+07 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 2 475 PCSetUpOnBlocks 61 1.0 6.8487e-02120.3 3.25e+07 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 2 475 PCApply 61 1.0 2.8346e+02 1.5 1.77e+09 1.0 7.8e+05 6.2e+03 2.2e+02 6 20 33 14 4 62 72 95 76 30 490 --- Event Stage 4: Corrector RHS Setup KSPGMRESOrthog 90 1.0 3.2514e+0113.8 2.01e+08 1.0 0.0e+00 0.0e+00 9.0e+01 1 2 0 0 1 4 10 0 0 4 491 KSPSetUp 24 1.0 4.3247e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+01 0 0 0 0 1 0 0 0 0 2 0 VecMDot 90 1.0 3.2177e+0115.2 1.00e+08 1.0 0.0e+00 0.0e+00 9.0e+01 1 1 0 0 1 4 5 0 0 4 248 VecNorm 99 1.0 1.6874e+00 2.8 2.01e+07 1.0 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 2 0 1 0 0 4 945 VecScale 99 1.0 7.6128e-02 2.9 1.00e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 10475 VecCopy 9 1.0 6.0771e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 159 1.0 2.5868e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 9 1.0 9.1977e-03 2.4 1.83e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 15763 VecMAXPY 99 1.0 5.1254e-01 1.8 1.19e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 6 0 0 0 18387 VecAssemblyBegin 123 1.0 2.9091e+0112.9 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+02 0 0 0 0 6 3 0 0 0 16 0 VecAssemblyEnd 123 1.0 3.9220e-04 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 99 1.0 1.9321e-01 3.2 1.00e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4127 VecScatterBegin 228 1.0 1.0614e-01 2.9 0.00e+00 0.0 2.0e+05 1.1e+04 0.0e+00 0 0 8 6 0 0 0 48 38 0 0 VecScatterEnd 228 1.0 1.3920e+02123.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 7 0 0 0 0 0 VecSetRandom 9 1.0 4.1017e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 99 1.0 1.7246e+00 2.7 3.01e+07 1.0 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 2 0 1 0 0 4 1387 MatMult 90 1.0 3.0840e+0147.6 1.62e+08 1.0 6.9e+04 8.0e+03 0.0e+00 0 2 3 2 0 1 8 17 10 0 410 MatConvert 9 1.0 1.7339e-01 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatScale 27 1.0 3.3595e+0016.6 2.37e+07 1.0 6.9e+03 8.0e+03 0.0e+00 0 0 0 0 0 0 1 2 1 0 552 MatAssemblyBegin 153 1.0 1.4866e+0212.2 0.00e+00 0.0 1.4e+04 2.0e+03 1.7e+02 3 0 1 0 3 16 0 3 0 7 0 MatAssemblyEnd 153 1.0 1.2535e+01 2.4 0.00e+00 0.0 8.3e+04 1.4e+03 4.3e+02 0 0 3 0 7 1 0 20 2 19 0 MatGetRow 1825272 1.0 7.0539e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCoarsen 9 1.0 6.8863e+01 1.0 0.00e+00 0.0 6.2e+04 1.2e+04 2.7e+02 2 0 3 2 4 11 0 15 14 12 0 MatAXPY 9 1.0 9.4477e-02 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMatMult 9 1.0 1.2057e+01 1.0 1.71e+07 1.0 4.4e+04 3.9e+03 2.2e+02 0 0 2 0 4 2 1 11 3 9 111 MatMatMultSym 9 1.0 8.4891e+00 1.0 0.00e+00 0.0 3.7e+04 3.2e+03 2.0e+02 0 0 2 0 3 1 0 9 2 9 0 MatMatMultNum 9 1.0 3.8710e+00 1.1 1.71e+07 1.0 6.9e+03 8.0e+03 1.8e+01 0 0 0 0 0 1 1 2 1 1 346 MatPtAP 9 1.0 4.6225e+01 1.0 3.24e+08 1.1 7.7e+04 8.0e+03 2.4e+02 1 4 3 2 4 7 15 19 11 10 538 MatPtAPSymbolic 9 1.0 3.3321e+01 1.0 0.00e+00 0.0 6.9e+04 6.8e+03 2.2e+02 1 0 3 1 4 5 0 17 8 10 0 MatPtAPNumeric 9 1.0 1.3878e+01 1.1 3.24e+08 1.1 7.9e+03 1.9e+04 1.8e+01 0 4 0 0 0 2 15 2 3 1 1793 MatTrnMatMult 9 1.0 2.6725e+02 1.0 1.35e+09 1.1 4.5e+04 5.7e+04 2.6e+02 7 15 2 7 4 41 63 11 46 11 390 MatGetLocalMat 45 1.0 6.0229e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 5.4e+01 0 0 0 0 1 0 0 0 0 2 0 MatGetBrAoCol 27 1.0 3.1634e+0123.3 0.00e+00 0.0 4.9e+04 1.1e+04 3.6e+01 0 0 2 2 1 2 0 12 10 2 0 MatGetSymTrans 18 1.0 5.0685e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 3 1.0 5.0362e+02 1.0 2.13e+09 1.1 4.0e+05 1.4e+04 2.1e+03 13 24 17 16 35 77100 97 99 91 329 PCGAMGgraph_AGG 3 1.0 3.2114e+00 1.2 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 0 0 2 0 2 139 PCGAMGcoarse_AGG 3 1.0 1.2110e+02 1.0 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 3 5 2 4 4 18 21 12 23 10 287 PCGAMGProl_AGG 3 1.0 1.0577e+01 1.3 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 1 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.5632e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 2 7 9 4 7 751 PCGAMGgraph_AGG 3 1.0 3.1994e+00 1.2 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 0 0 2 0 2 139 PCGAMGcoarse_AGG 3 1.0 1.3373e+02 1.1 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 3 5 2 4 4 20 21 12 23 10 260 PCGAMGProl_AGG 3 1.0 1.3093e+01 2.1 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 2 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.5847e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 2 7 9 4 7 741 PCGAMGgraph_AGG 3 1.0 3.1866e+00 1.2 5.70e+06 1.0 6.9e+03 4.0e+03 5.7e+01 0 0 0 0 1 0 0 2 0 2 140 PCGAMGcoarse_AGG 3 1.0 1.2222e+02 1.0 4.49e+08 1.1 4.9e+04 2.6e+04 2.4e+02 3 5 2 4 4 19 21 12 23 10 284 PCGAMGProl_AGG 3 1.0 1.0587e+01 1.3 0.00e+00 0.0 1.4e+04 6.3e+03 8.4e+01 0 0 1 0 1 1 0 3 2 4 0 PCGAMGPOpt_AGG 3 1.0 1.6119e+01 1.0 1.49e+08 1.0 3.8e+04 6.4e+03 1.6e+02 0 2 2 1 3 2 7 9 4 7 729 --- Event Stage 5: Corrector Solve KSPGMRESOrthog 54 1.0 1.3972e-01 3.5 9.35e+04 0.0 0.0e+00 0.0e+00 5.4e+01 0 0 0 0 1 0 0 0 0 9 1 KSPSetUp 3 1.0 4.0531e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 3 1.0 3.3334e+02 1.0 2.16e+09 1.0 7.3e+05 7.8e+03 6.3e+02 9 24 30 16 10 100100100100100 511 VecMDot 54 1.0 1.3947e-01 3.5 4.67e+04 0.0 0.0e+00 0.0e+00 5.4e+01 0 0 0 0 1 0 0 0 0 9 0 VecTDot 102 1.0 3.7865e+0111.1 5.64e+07 1.0 0.0e+00 0.0e+00 1.0e+02 1 1 0 0 2 9 3 0 0 16 119 VecNorm 219 1.0 1.2941e+0213.7 6.15e+07 1.0 0.0e+00 0.0e+00 2.2e+02 2 1 0 0 4 24 3 0 0 35 38 VecScale 108 1.0 1.5640e-04 4.5 4.68e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 299 VecCopy 276 1.0 2.6649e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 765 1.0 1.6780e-01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 480 1.0 8.1834e-01 2.7 1.22e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 6 0 0 0 11892 VecAYPX 750 1.0 7.2687e-01 2.0 1.07e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 5 0 0 0 11745 VecMAXPY 108 1.0 1.9693e-04 5.8 9.35e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 475 VecAssemblyBegin 3 1.0 1.5276e-01 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 1 0 VecAssemblyEnd 3 1.0 2.5034e-05 4.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecPointwiseMult 324 1.0 5.8169e-01 2.6 3.29e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 4486 VecScatterBegin 1131 1.0 3.2379e-01 2.8 0.00e+00 0.0 7.3e+05 7.8e+03 0.0e+00 0 0 30 16 0 0 0100100 0 0 VecScatterEnd 1131 1.0 3.1012e+02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 64 0 0 0 0 0 VecNormalize 108 1.0 5.9516e+00124.0 1.40e+05 0.0 0.0e+00 0.0e+00 1.1e+02 0 0 0 0 2 1 0 0 0 17 0 MatMult 807 1.0 2.8924e+02 4.6 1.54e+09 1.0 5.4e+05 9.9e+03 0.0e+00 4 17 22 15 0 46 71 73 94 0 419 MatMultAdd 162 1.0 2.5926e+01 3.9 1.18e+08 1.0 9.7e+04 1.7e+03 0.0e+00 0 1 4 0 0 5 5 13 3 0 360 MatMultTranspose 162 1.0 8.3483e+01 5.0 1.18e+08 1.0 9.7e+04 1.7e+03 0.0e+00 1 1 4 0 0 15 5 13 3 0 112 MatSolve 108 0.0 2.8933e-02 0.0 1.90e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 657 MatLUFactorSym 3 1.0 1.0177e-02212.4 0.00e+00 0.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 0 0 0 1 0 MatLUFactorNum 3 1.0 5.5764e-0219490.9 3.25e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 583 MatGetRowIJ 3 0.0 7.7796e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 3 0.0 2.1908e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.5e-02 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 3 1.0 6.8519e-02135.6 3.25e+07 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 2 475 PCSetUpOnBlocks 54 1.0 6.8580e-02122.7 3.25e+07 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 0 0 0 0 0 2 474 PCApply 54 1.0 2.4933e+02 1.5 1.57e+09 1.0 6.9e+05 6.2e+03 2.0e+02 5 18 29 12 3 62 72 95 76 31 493 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Container 0 6 3288 0 Krylov Solver 0 37 145744 0 Vector 7 242 208227912 0 Vector Scatter 0 70 72520 0 Matrix 0 216 519411504 0 Distributed Mesh 0 14 48780296 0 Bipartite Graph 0 28 19152 0 Index Set 0 30 43104 0 IS L to G Mapping 0 21 32484340 0 Preconditioner 0 36 35112 0 --- Event Stage 1: DMMG Setup Krylov Solver 1 0 0 0 Vector 5 4 5856 0 Vector Scatter 4 0 0 0 Distributed Mesh 2 0 0 0 Bipartite Graph 4 0 0 0 Index Set 10 10 2326068 0 IS L to G Mapping 3 0 0 0 --- Event Stage 2: Pressure RHS Setup Container 3 0 0 0 Krylov Solver 27 9 271080 0 Vector 501 393 188495712 0 Vector Scatter 78 45 46620 0 Matrix 258 153 815233956 0 Matrix Coarsen 9 9 5508 0 Distributed Mesh 6 0 0 0 Bipartite Graph 12 0 0 0 Index Set 186 180 8080116 0 IS L to G Mapping 9 0 0 0 Preconditioner 27 9 7704 0 Viewer 1 0 0 0 PetscRandom 9 9 5472 0 --- Event Stage 3: Pressure Solve Vector 131 122 270026992 0 Matrix 3 0 0 0 Index Set 9 0 0 0 --- Event Stage 4: Corrector RHS Setup Container 3 0 0 0 Krylov Solver 27 9 271080 0 Vector 501 393 188495712 0 Vector Scatter 78 45 46620 0 Matrix 258 153 815233956 0 Matrix Coarsen 9 9 5508 0 Distributed Mesh 6 0 0 0 Bipartite Graph 12 0 0 0 Index Set 186 180 8080116 0 IS L to G Mapping 9 0 0 0 Preconditioner 27 9 7704 0 PetscRandom 9 9 5472 0 --- Event Stage 5: Corrector Solve Vector 117 108 239040288 0 Matrix 3 0 0 0 Index Set 9 0 0 0 ======================================================================================================================== Average time to get PetscTime(): 1.90735e-07 Average time for MPI_Barrier(): 0.000685406 Average time for zero size MPI_Send(): 0.000561473 #PETSc Option Table entries: -ksp_converged_reason -ksp_monitor_true_residual -ksp_rtol 1.0e-7 -ksp_type cg -log_summary -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -pc_gamg_agg_nsmooths 1 -pc_type gamg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure run at: Tue Sep 18 16:22:39 2012 Configure options: --with-debugging=0 --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich ----------------------------------------- Libraries compiled on Tue Sep 18 16:22:39 2012 on firefox.bioinfo.ittc.ku.edu Machine characteristics: Linux-2.6.18-92.1.13.el5-x86_64-with-redhat-5.2-Final Using PETSc directory: /bio/work1/zlwei/PETSc/petsc-dev Using PETSc arch: arch-linux2-c-debug ----------------------------------------- Using C compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS} Using Fortran compiler: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS} ----------------------------------------- Using include paths: -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/include ----------------------------------------- Using C linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpicc Using Fortran linker: /bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/bin/mpif90 Using libraries: -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/arch-linux2-c-debug/lib -lflapack -lfblas -lm -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lgcc_s -ldl ----------------------------------------- From knepley at gmail.com Mon Oct 22 11:31:40 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Oct 2012 12:31:40 -0400 Subject: [petsc-users] Need a diagnose for the slow performance In-Reply-To: <508570F8.2020406@gmail.com> References: <508570F8.2020406@gmail.com> Message-ID: On Mon, Oct 22, 2012 at 12:14 PM, Zhenglun (Alan) Wei < zhenglun.wei at gmail.com> wrote: > Dear all, > I hope you're having a nice day. > I need some help to figure out what is wrong in my code. I had very > good performance after I change the preconditioner. The out_1, which is > attached, shows that. Last week, in order to use PETSc in out new cluster, > I reconfigure the PETSc with the same configuration in the new cluster and > saved to another directory. My code can obtain very good performance in the > new machine. However, my code performs very bad in the old cluster, as > shown in out_2 in the attachment. > Does any one have any idea on this? > The solver is obviously different since it takes vastly different numbers of iterates. Matt > thanks, > Alan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.spott at gmail.com Mon Oct 22 17:18:15 2012 From: andrew.spott at gmail.com (Andrew Spott) Date: Mon, 22 Oct 2012 16:18:15 -0600 Subject: [petsc-users] Determining the correct Preallocation Message-ID: I want to preallocate a MPIAIJ matrix correctly, using MatMPIAIJSetPreallocation(A, PETSC_NULL, dnnz, PETSC_NULL, onnz); The problem is that I don't know how many local rows, or where the local columns are until I run said function, but I need to know the number of local rows and local columns so I can determine the correct preallocation. So how am I supposed to do a proper preallocation? Is there something I'm missing, or do I just guess and correct? If I know that my matrix is evenly sparse, should I just assume that it is evenly split up among processors, and go from there? Or should I "MatSetUp", determine the correct preallocation and then do a MatMPIAIJSetPreallocation? -Andrew From knepley at gmail.com Mon Oct 22 17:20:55 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Oct 2012 18:20:55 -0400 Subject: [petsc-users] Determining the correct Preallocation In-Reply-To: References: Message-ID: On Mon, Oct 22, 2012 at 6:18 PM, Andrew Spott wrote: > I want to preallocate a MPIAIJ matrix correctly, using > > MatMPIAIJSetPreallocation(A, PETSC_NULL, dnnz, PETSC_NULL, onnz); > > The problem is that I don't know how many local rows, or where the local > columns are until I run said function, but I need to know the number of > local rows and local columns so I can determine the correct preallocation. > > So how am I supposed to do a proper preallocation? Is there something I'm > missing, or do I just guess and correct? If I know that my matrix is > evenly sparse, should I just assume that it is evenly split up among > processors, and go from there? Or should I "MatSetUp", determine the > correct preallocation and then do a MatMPIAIJSetPreallocation? I think you are looking for http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscSplitOwnership.html If you do not specify the layout, this function is called to partition your matrix. Thanks, Matt > -Andrew -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Oct 23 01:16:58 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 23 Oct 2012 02:16:58 -0400 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ Message-ID: Hi PETSc Users, I have to subtract a submatrix from a blocked parallel matrix (MPIBAIJ). from K_11 K_12 to get K_11 K_21 K_22 The code is call ISCreateStride(PETSC_COMM_WORLD,length,0,1,isrowpp,ierr) call MatGetSubMatrices(Kmatem,1,isrowpp,isrowpp, MAT_INITIAL_MATRIX,Kppseq,ierr) When I am running it, the command indicates as [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Indices are not block ordered! [0]PETSC ERROR: ------------------------------------------------------------------------ The reason seems pretty clear that the IS need to be in block format either. Although the matrix is created as blocked, during the calculation I have already permute the indices so that what I need from the matrix is the square matrix exactly from 0--length. Can anyone tell me based on my demand, how should I create this blocked IS to get submatrix? Thank you very much! Best, Shu From jedbrown at mcs.anl.gov Tue Oct 23 06:46:03 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 06:46:03 -0500 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 1:16 AM, shu guo wrote: > Hi PETSc Users, > > I have to subtract a submatrix from a blocked parallel matrix (MPIBAIJ). > > from K_11 K_12 to get K_11 > K_21 K_22 > > The code is > > call ISCreateStride(PETSC_COMM_WORLD,length,0,1,isrowpp,ierr) > call MatGetSubMatrices(Kmatem,1,isrowpp,isrowpp, > MAT_INITIAL_MATRIX,Kppseq,ierr) > > When I am running it, the command indicates as > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Arguments are incompatible! > [0]PETSC ERROR: Indices are not block ordered! > Always send the ENTIRE error message. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > > > The reason seems pretty clear that the IS need to be in block format > either. Although the matrix is created as blocked, during the > calculation I have already permute the indices Why are you doing this permutation? How does it look in parallel? > so that what I need > from the matrix is the square matrix exactly from 0--length. Can > anyone tell me based on my demand, how should I create this blocked IS > to get submatrix? Thank you very much! > Best, > Shu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzj at umich.edu Tue Oct 23 08:49:16 2012 From: lzj at umich.edu (Peter Zhoujie Lyu) Date: Tue, 23 Oct 2012 09:49:16 -0400 Subject: [petsc-users] VecView and MatView Question Message-ID: Hi everyone, My code dumps petsc vec and mat into an output file using VecView and MatView. I was wondering if there is any way to set the format, such as number of digits that was showing in the output file? call PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) call PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) call VecView(dJdW,viewer,PETScIerr) Thanks, Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 23 08:51:45 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Oct 2012 09:51:45 -0400 Subject: [petsc-users] VecView and MatView Question In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 9:49 AM, Peter Zhoujie Lyu wrote: > Hi everyone, > > My code dumps petsc vec and mat into an output file using VecView and > MatView. I was wondering if there is any way to set the format, such as > number of digits that was showing in the output file? > > call > PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) > call PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) > call VecView(dJdW,viewer,PETScIerr) > There is not. Our intention for ASCII output is debugging only. For production, we suggest using the binary output which is smaller and faster, or perhaps the HDF5 for which many good tools exist. Thanks, Matt > Thanks, > > Peter > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From lzj at umich.edu Tue Oct 23 09:08:08 2012 From: lzj at umich.edu (Peter Zhoujie Lyu) Date: Tue, 23 Oct 2012 10:08:08 -0400 Subject: [petsc-users] VecView and MatView Question In-Reply-To: References: Message-ID: Okay, thanks. My output file is for debugging purpose. I need to check the possible errors at much higher digits, where VecView cut off at 6 digits for my case. I will try out HDF5 output then. Thanks, Peter On Tue, Oct 23, 2012 at 9:51 AM, Matthew Knepley wrote: > On Tue, Oct 23, 2012 at 9:49 AM, Peter Zhoujie Lyu wrote: > >> Hi everyone, >> >> My code dumps petsc vec and mat into an output file using VecView and >> MatView. I was wondering if there is any way to set the format, such as >> number of digits that was showing in the output file? >> >> call >> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) >> call PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) >> call VecView(dJdW,viewer,PETScIerr) >> > > There is not. Our intention for ASCII output is debugging only. For > production, we suggest using > the binary output which is smaller and faster, or perhaps the HDF5 for > which many good tools exist. > > Thanks, > > Matt > > >> Thanks, >> >> Peter >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lvzhoujie at gmail.com Tue Oct 23 09:07:00 2012 From: lvzhoujie at gmail.com (Peter Zhoujie Lyu) Date: Tue, 23 Oct 2012 10:07:00 -0400 Subject: [petsc-users] VecView and MatView Question In-Reply-To: References: Message-ID: Okay, thanks. My output file is for debugging purpose. I need to check the possible errors at much higher digits, where VecView cut off at 6 digits for my case. I will try out HDF5 output then. Thanks, Peter On Tue, Oct 23, 2012 at 9:51 AM, Matthew Knepley wrote: > On Tue, Oct 23, 2012 at 9:49 AM, Peter Zhoujie Lyu wrote: > >> Hi everyone, >> >> My code dumps petsc vec and mat into an output file using VecView and >> MatView. I was wondering if there is any way to set the format, such as >> number of digits that was showing in the output file? >> >> call >> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) >> call PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) >> call VecView(dJdW,viewer,PETScIerr) >> > > There is not. Our intention for ASCII output is debugging only. For > production, we suggest using > the binary output which is smaller and faster, or perhaps the HDF5 for > which many good tools exist. > > Thanks, > > Matt > > >> Thanks, >> >> Peter >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Oct 23 09:56:21 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 23 Oct 2012 10:56:21 -0400 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: Hi Jed, Thank you for your reply. I have figured out how this error happens. When I get a sub-matrix from a block matrix, Petsc will compress the IS according to the matrix block format. So for example, if the matrix has a block size nsbk=4. It will calculate the dimension/nsbk and sort IS accordingly. The original matrix is like A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 so each node has 4 degree of freedom. And the way I permute the matrix is sort the matrix like, A1 A2 || B1_1 B1_2 B1_3 B2_1 B2_2 B2_3 So I need the sub-matrix A1 A2. Can you give me a hint how to do it? Another question is if I do not permute the matrix, so it will hold the format of A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 If I want to get sub-matrix A1 A2 from it. I need a blocked IS, right? In the manual, I found a statement that saying "For BAIJ matrices the index sets must respect the block structure, that is if they request one row/column in a block, they must request all rows/columns that are in that block. For example, if the block size is 2 you cannot request just row 0 and column 0." Does it mean that I cannot merely access A1 in the first block? If not, how should I do with this structure if I want A1 A2 from it? Thank you very much! Best, Shu On Tue, Oct 23, 2012 at 7:46 AM, Jed Brown wrote: > On Tue, Oct 23, 2012 at 1:16 AM, shu guo wrote: >> >> Hi PETSc Users, >> >> I have to subtract a submatrix from a blocked parallel matrix (MPIBAIJ). >> >> from K_11 K_12 to get K_11 >> K_21 K_22 >> >> The code is >> >> call ISCreateStride(PETSC_COMM_WORLD,length,0,1,isrowpp,ierr) >> call MatGetSubMatrices(Kmatem,1,isrowpp,isrowpp, >> MAT_INITIAL_MATRIX,Kppseq,ierr) >> >> When I am running it, the command indicates as >> >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Arguments are incompatible! >> [0]PETSC ERROR: Indices are not block ordered! > > > Always send the ENTIRE error message. > >> >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ >> >> >> The reason seems pretty clear that the IS need to be in block format >> either. Although the matrix is created as blocked, during the >> calculation I have already permute the indices > > > Why are you doing this permutation? How does it look in parallel? > >> >> so that what I need >> from the matrix is the square matrix exactly from 0--length. Can >> anyone tell me based on my demand, how should I create this blocked IS >> to get submatrix? Thank you very much! >> >> >> Best, >> Shu > > From jedbrown at mcs.anl.gov Tue Oct 23 10:04:11 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 10:04:11 -0500 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 9:56 AM, shu guo wrote: > I have figured out how this error happens. When I get a sub-matrix > from a block matrix, Petsc will compress the IS according to the > matrix block format. So for example, if the matrix has a block size > nsbk=4. It will calculate the dimension/nsbk and sort IS accordingly. > > The original matrix is like > > A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 > I don't understand this notation. > > so each node has 4 degree of freedom. And the way I permute the matrix > is sort the matrix like, > > A1 A2 || B1_1 B1_2 B1_3 B2_1 B2_2 B2_3 > *Why* are you permuting the matrix like this? Why not just construct the index set that addresses the part you want? > > So I need the sub-matrix A1 A2. Can you give me a hint how to do it? > > Another question is if I do not permute the matrix, so it will hold > the format of > A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 > > If I want to get sub-matrix A1 A2 from it. I need a blocked IS, right? > In the manual, I found a statement that saying "For BAIJ matrices the > index sets must respect the block structure, that is if they request > one row/column in a block, they must request all rows/columns that are > in that block. For example, if the block size is 2 you cannot request > just row 0 and column 0." Does it mean that I cannot merely access A1 > in the first block? If not, how should I do with this structure if I > want A1 A2 from it? > Just build an index set that has every part of the block. If you need arbitrary scalar subsets, use MPIAIJ (can use MatConvert). -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Oct 23 10:19:36 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 23 Oct 2012 11:19:36 -0400 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: Hi Jed, I am solving a electromagnetic problem. Each node has 4 degree of freedom, phi A1 A2 A3. The row in this blocked matrix is generated as node1 | node 2 | node 3 | | phi1 A1_1 A1_2 A1_3 | phi2 A2_1 A2_2 A2_3| phi3 A3_1 A3_2 A3_3 And I permute it as phi matrix || A matrix phi1 phi2 phi3 || A1_1 A1_2 A1_3 A2_1 A2_2 A2_3 A3_1 A3_2 A3_3 I permuted it just want to sort each variables together. > Just build an index set that has every part of the block. If you need > arbitrary scalar subsets, use MPIAIJ (can use MatConvert). If I want the submatrix from original matrix, can you be more specific that how the index set looks like? And is it possible for me to merely get the submatrix of the ist DOF of each element? Thank you. Shu On Tue, Oct 23, 2012 at 11:04 AM, Jed Brown wrote: > On Tue, Oct 23, 2012 at 9:56 AM, shu guo wrote: >> >> I have figured out how this error happens. When I get a sub-matrix >> from a block matrix, Petsc will compress the IS according to the >> matrix block format. So for example, if the matrix has a block size >> nsbk=4. It will calculate the dimension/nsbk and sort IS accordingly. >> >> The original matrix is like >> >> A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 > > > I don't understand this notation. > >> >> >> so each node has 4 degree of freedom. And the way I permute the matrix >> is sort the matrix like, >> >> A1 A2 || B1_1 B1_2 B1_3 B2_1 B2_2 B2_3 > > > Why are you permuting the matrix like this? Why not just construct the index > set that addresses the part you want? > >> >> >> So I need the sub-matrix A1 A2. Can you give me a hint how to do it? >> >> Another question is if I do not permute the matrix, so it will hold >> the format of >> A1 B1_1 B1_2 B1_3 || A2 B2_1 B2_2 B2_3 >> >> If I want to get sub-matrix A1 A2 from it. I need a blocked IS, right? >> In the manual, I found a statement that saying "For BAIJ matrices the >> index sets must respect the block structure, that is if they request >> one row/column in a block, they must request all rows/columns that are >> in that block. For example, if the block size is 2 you cannot request >> just row 0 and column 0." Does it mean that I cannot merely access A1 >> in the first block? If not, how should I do with this structure if I >> want A1 A2 from it? > > > Just build an index set that has every part of the block. If you need > arbitrary scalar subsets, use MPIAIJ (can use MatConvert). From jedbrown at mcs.anl.gov Tue Oct 23 10:26:26 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 10:26:26 -0500 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 10:19 AM, shu guo wrote: > I am solving a electromagnetic problem. Each node has 4 degree of > freedom, phi A1 A2 A3. > > The row in this blocked matrix is generated as > > node1 | node 2 | node 3 > | | > phi1 A1_1 A1_2 A1_3 | phi2 A2_1 A2_2 A2_3| phi3 A3_1 A3_2 A3_3 > > And I permute it as > > phi matrix || A matrix > phi1 phi2 phi3 || A1_1 A1_2 A1_3 A2_1 A2_2 A2_3 A3_1 A3_2 A3_3 > > I permuted it just want to sort each variables together. > Why? This just destroys the locality that makes BAIJ efficient. See the effect of "field interlacing" here (still true today). http://www.mcs.anl.gov/~kaushik/Papers/parco01.pdf > > > Just build an index set that has every part of the block. If you need > > arbitrary scalar subsets, use MPIAIJ (can use MatConvert). > > If I want the submatrix from original matrix, can you be more specific > that how the index set looks like? And is it possible for me to merely > get the submatrix of the ist DOF of each element? Thank you. > Create an index set with [0,4,8,...], but use the AIJ format. You can request support for BAIJ here, but I'm afraid it's not going to be considered high priority because the potential benefits of BAIJ over AIJ are quite small if you are taking submatrices. https://bitbucket.org/petsc/petsc-dev/issues -------------- next part -------------- An HTML attachment was scrubbed... URL: From greatgs2008 at gmail.com Tue Oct 23 11:12:01 2012 From: greatgs2008 at gmail.com (shu guo) Date: Tue, 23 Oct 2012 12:12:01 -0400 Subject: [petsc-users] MatGetSubmatrices from block matrix MPIBAIJ In-Reply-To: References: Message-ID: Hi Jed, Thank you for the paper. It is very interesting and helpful. Shu On Tue, Oct 23, 2012 at 11:26 AM, Jed Brown wrote: > On Tue, Oct 23, 2012 at 10:19 AM, shu guo wrote: >> >> I am solving a electromagnetic problem. Each node has 4 degree of >> freedom, phi A1 A2 A3. >> >> The row in this blocked matrix is generated as >> >> node1 | node 2 | node >> 3 >> | | >> phi1 A1_1 A1_2 A1_3 | phi2 A2_1 A2_2 A2_3| phi3 A3_1 A3_2 A3_3 >> >> And I permute it as >> >> phi matrix || A matrix >> phi1 phi2 phi3 || A1_1 A1_2 A1_3 A2_1 A2_2 A2_3 A3_1 A3_2 A3_3 >> >> I permuted it just want to sort each variables together. > > > Why? This just destroys the locality that makes BAIJ efficient. See the > effect of "field interlacing" here (still true today). > > http://www.mcs.anl.gov/~kaushik/Papers/parco01.pdf > >> >> >> > Just build an index set that has every part of the block. If you need >> > arbitrary scalar subsets, use MPIAIJ (can use MatConvert). >> >> If I want the submatrix from original matrix, can you be more specific >> that how the index set looks like? And is it possible for me to merely >> get the submatrix of the ist DOF of each element? Thank you. > > > Create an index set with [0,4,8,...], but use the AIJ format. > > You can request support for BAIJ here, but I'm afraid it's not going to be > considered high priority because the potential benefits of BAIJ over AIJ are > quite small if you are taking submatrices. > https://bitbucket.org/petsc/petsc-dev/issues From bsmith at mcs.anl.gov Tue Oct 23 12:49:51 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Tue, 23 Oct 2012 12:49:51 -0500 Subject: [petsc-users] VecView and MatView Question In-Reply-To: References: Message-ID: <535F3D67-A0A9-4F2D-BF8C-A55C4911692D@mcs.anl.gov> Peter, You can use PETSC_VIEWER_ASCII_MATLAB this causes all the decimal digits to be printed (useful for debugging) Barry Just ignore the fact that it is related to MATLAB. On Oct 23, 2012, at 9:08 AM, Peter Zhoujie Lyu wrote: > Okay, thanks. My output file is for debugging purpose. I need to check the possible errors at much higher digits, where VecView cut off at 6 digits for my case. I will try out HDF5 output then. > > Thanks, > > Peter > > On Tue, Oct 23, 2012 at 9:51 AM, Matthew Knepley wrote: > On Tue, Oct 23, 2012 at 9:49 AM, Peter Zhoujie Lyu wrote: > Hi everyone, > > My code dumps petsc vec and mat into an output file using VecView and MatView. I was wondering if there is any way to set the format, such as number of digits that was showing in the output file? > > call PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) > call PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) > call VecView(dJdW,viewer,PETScIerr) > > There is not. Our intention for ASCII output is debugging only. For production, we suggest using > the binary output which is smaller and faster, or perhaps the HDF5 for which many good tools exist. > > Thanks, > > Matt > > Thanks, > > Peter > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > From jzhong at scsolutions.com Tue Oct 23 13:03:07 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 18:03:07 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc Message-ID: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> Dear folks, I have a question on how to use mumps properly in PETSc. It appears that I didn?t set up mumps right. I followed the example in http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html to set up my program. Here is my situation on using the default setting PetscInt icntl_7 = 5; MatMumpsSetIcntl(F,7,icntl_7); in the example ex125.c: 1. The program work fine on all small models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3. 2. The program has some issues on medium size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB. a. We used another software to generate sparse matrix by using 144 cores: i. When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. b. We used another software to generate the same sparse matrix by using 576 cores: i. When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. 3. The program could not solve the large size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix). The full sparse matrix is sized at 511GB. a. We used another software to generate sparse matrix by using 900 cores: i. When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT provide the solution. My confusion starts from the medium size problem: ? It seems something was not right in the default setting in ex125.c for these problems. ? I got the info that METIS was used instead of ParMETIS in solving these problems. ? Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem. ? I suspect one rank was trying to collect all data from other ranks. What other addition setting is needed for mumps such that it could deal with medium and large size problems? Do you guys have similar experience on that? Thanks, Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 13:55:27 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 13:55:27 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 1:03 PM, Jinquan Zhong wrote: > Dear folks,**** > > ** ** > > I have a question on how to use mumps properly in PETSc. It appears that > I didn?t set up mumps right. I followed the example in **** > > http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html > **** > > ** ** > > to set up my program. Here is my situation on using the default setting * > *** > > ** ** > > PetscInt icntl_7 = 5;**** > > MatMumpsSetIcntl > (F,7,icntl_7);**** > > ** ** > > in the example ex125.c: **** > > ** ** > > **1. **The program work fine on all *small* models (sparse matrices > at the order of m= 894, 1097, 31k with a dense matrix included in the > sparse matrix). The residuals are at the magnitude of 10^-3. > This suggests that your systems are nearly singular. If you have a condition number of 1e12, it's time to reconsider the model. > **** > > ** ** > > **2. **The program has some issues on *medium* size problem > (m=460k with a dense matrix at the order of n=30k included in the sparse > matrix). The full sparse matrix is sized at 17GB.**** > > **a. **We used another software to generate sparse matrix by using > 144 cores: **** > > ** i. * > *When I used the resource from 144 cores (12 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation. > Always send the entire error message. > **** > > ** ii. **When > I used the resource from 432 cores (36 nodes with 48GB/node), it provided > the solution. **** > > **b. **We used another software to generate the same sparse matrix > by using 576 cores: **** > > ** i. * > *When I used the resource from 576 cores (48 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation.**** > > ** ii. **When > I used the resource from 1152 cores (96 nodes with 48GB/node), it provided > the solution. **** > > ** ** > > **3. **The program could not solve the *large* size problem (m=640k > with a dense matrix at the order of n=178k included in the sparse matrix). > The full sparse matrix is sized at 511GB.**** > > **a. **We used another software to generate sparse matrix by using > 900 cores: **** > > ** i. * > *When I used the resource from 900 cores (75 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation.**** > > ** ii. **When > I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL > COULD NOT provide the solution. > This has a huge dense block and we can't tell from your description how large the vertex separators are. MUMPS is well-known to have some non-scalable data structures. They do some analysis on rank 0 and require right hand sides to be provided entirely on rank 0. > **** > > ** ** > > My confusion starts from the medium size problem:**** > > **? **It seems something was not right in the default setting in > ex125.c for these problems. > I don't know what you're asking. Do a heap profile if you want to find out where the memory is leaking. It's *much* better to call the solver directly from the process that assembles the matrix. Going through a file is terribly wasteful. > **** > > **? **I got the info that METIS was used instead of ParMETIS in > solving these problems. > Did you ask for parallel ordering (-mat_mumps_icntl_28) and parmetis (-mat_mumps_icntl_29)? These options are shown in -help. (It's not our fault the MUMPS developers have Fortran numbered option insanity baked in. Read their manual and translate to our numbered options. While you're at it, ask them to write a decent options and error reporting mechanism. > **** > > **? **Furthermore, it appears that there was unreasonable demand > on the solver even on the medium size problem. > Good, it's easier to debug. Check -log_summary for that run and use the heap profilers at your computing facility. > **** > > **? **I suspect one rank was trying to collect all data from > other ranks. > Naturally. > **** > > ** ** > > What other addition setting is needed for mumps such that it could deal > with medium and large size problems?**** > > ** ** > > Do you guys have similar experience on that?**** > > ** ** > > Thanks,**** > > ** ** > > Jinquan**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Oct 23 14:02:51 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 23 Oct 2012 14:02:51 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> Message-ID: Jinquan: > > I have a question on how to use mumps properly in PETSc. It appears that > I didn?t set up mumps right. I followed the example in **** > > http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html > This example is for our internal testing, not intended for production runs. Suggest using PETSc high level KSP solver which provides more flexibility. > **** > > ** ** > > to set up my program. Here is my situation on using the default setting * > *** > > ** ** > > PetscInt icntl_7 = 5;**** > > MatMumpsSetIcntl > (F,7,icntl_7);**** > > ** ** > > in the example ex125.c: > Using KSP sover, the mumps options can be chosen at runtime, e.g. ~petsc/src/ksp/ksp/examples/tutorials/ex2.c: mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_7 5 Norm of error 1.49777e-15 iterations 1 > **1. **The program work fine on all *small* models (sparse matrices > at the order of m= 894, 1097, 31k with a dense matrix included in the > sparse matrix). The residuals are at the magnitude of 10^-3. > ^^^^ With a direct solver, residual = 10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ? Is this the reason you want to use a direct solver instead of iterative one? What do you mean "31k with a dense matrix included in the sparse matrix"? How sparse is your matrix, e.g., nnz(A)/(m*m)=? > **2. **The program has some issues on *medium* size problem > (m=460k with a dense matrix at the order of n=30k included in the sparse > matrix). The full sparse matrix is sized at 17GB.**** > > **a. **We used another software to generate sparse matrix by using > 144 cores: **** > > ** i. * > *When I used the resource from 144 cores (12 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation.**** > > ** ii. **When > I used the resource from 432 cores (36 nodes with 48GB/node), it provided > the solution. > Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. What is "another software "? > **** > > **b. **We used another software to generate the same sparse matrix > by using 576 cores: **** > > ** i. * > *When I used the resource from 576 cores (48 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation.**** > > ** ii. **When > I used the resource from 1152 cores (96 nodes with 48GB/node), it provided > the solution. > Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b. This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c mpiexec -n 2 ./ex10 -f -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view ... then you'll see memory info provided by mumps. > **3. **The program could not solve the *large* size problem (m=640k > with a dense matrix at the order of n=178k included in the sparse matrix). > The full sparse matrix is sized at 511GB.**** > > **a. **We used another software to generate sparse matrix by using > 900 cores: **** > > ** i. * > *When I used the resource from 900 cores (75 nodes with 48GB/node), it > could not provide the solution. There was a complain on the memory > violation.**** > > ** ii. **When > I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL > COULD NOT provide the solution. > Your computer system and software have limits. Find the answers to your 'medium size' problems first. > **** > > > My confusion starts from the medium size problem:**** > > **? **It seems something was not right in the default setting in > ex125.c for these problems. **** > > **? **I got the info that METIS was used instead of ParMETIS in > solving these problems. > By default, petsc-mumps interface uses sequential symbolic factorization (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel. I tested it, but seems parmetis is still not used. Check mumps manual or contact mumps developer on how to use parmetis. > **** > > **? **Furthermore, it appears that there was unreasonable demand > on the solver even on the medium size problem. **** > > **? **I suspect one rank was trying to collect all data from > other ranks. > Yes, analysis is sequential, and rhs vector must be in the host :-( In general, direct solvers cannot be scaled to very large num of cores. > **** > > ** ** > > What other addition setting is needed for mumps such that it could deal > with medium and large size problems? > Run your code with option '-help |grep mumps' and experiment with various options, e.g., matrix orderings, nonzero-fills etc. You may also try superlu_dist. Good luck! > **** > > ** ** > > Do you guys have similar experience on that? > I personally never used mumps or superlu_dist for such large matrices. Consult mumps developers. Hong > **** > > ** ** > > Thanks,**** > > ** ** > > Jinquan**** > > ** ** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lvzhoujie at gmail.com Tue Oct 23 14:15:54 2012 From: lvzhoujie at gmail.com (Peter Zhoujie Lyu) Date: Tue, 23 Oct 2012 15:15:54 -0400 Subject: [petsc-users] VecView and MatView Question In-Reply-To: <535F3D67-A0A9-4F2D-BF8C-A55C4911692D@mcs.anl.gov> References: <535F3D67-A0A9-4F2D-BF8C-A55C4911692D@mcs.anl.gov> Message-ID: Great! That works. Thanks a lot. Peter On Tue, Oct 23, 2012 at 1:49 PM, Barry Smith wrote: > > Peter, > > You can use PETSC_VIEWER_ASCII_MATLAB this causes all the decimal > digits to be printed (useful for debugging) > > Barry > > > Just ignore the fact that it is related to MATLAB. > > > On Oct 23, 2012, at 9:08 AM, Peter Zhoujie Lyu wrote: > > > Okay, thanks. My output file is for debugging purpose. I need to check > the possible errors at much higher digits, where VecView cut off at 6 > digits for my case. I will try out HDF5 output then. > > > > Thanks, > > > > Peter > > > > On Tue, Oct 23, 2012 at 9:51 AM, Matthew Knepley > wrote: > > On Tue, Oct 23, 2012 at 9:49 AM, Peter Zhoujie Lyu > wrote: > > Hi everyone, > > > > My code dumps petsc vec and mat into an output file using VecView and > MatView. I was wondering if there is any way to set the format, such as > number of digits that was showing in the output file? > > > > call > PetscViewerASCIIOpen(PETSC_COMM_WORLD,"dJdWd.output",viewer,PETScIerr) > > call > PetscViewerSetFormat(viewer,PETSC_VIEWER_ASCII_INDEX,PetscIerr) > > call VecView(dJdW,viewer,PETScIerr) > > > > There is not. Our intention for ASCII output is debugging only. For > production, we suggest using > > the binary output which is smaller and faster, or perhaps the HDF5 for > which many good tools exist. > > > > Thanks, > > > > Matt > > > > Thanks, > > > > Peter > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 14:36:14 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 19:36:14 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> Thanks, Hong and Jed. 1. The program work fine on all small models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3. ^^^^ With a direct solver, residual = 10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ? >> That is a good point. In these applications, we don't usually have a good-conditioned matrix . The condition number is always around 10^10--10^12. This is out of our control. Is this the reason you want to use a direct solver instead of iterative one? What do you mean "31k with a dense matrix included in the sparse matrix"? >> We have a dense matrix embedded inside a sparse matrix. This dense matrix usually accounts for 99% of the total nnz's. How sparse is your matrix, e.g., nnz(A)/(m*m)=? >> ~=0.4% for medium size problem and 0.04% for large size problem. 2. The program has some issues on medium size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB. a. We used another software to generate sparse matrix by using 144 cores: i. When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. What is "another software "? >> it is a propriety software that I don't have access to. b. We used another software to generate the same sparse matrix by using 576 cores: i. When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b. >> My confusion is that since the sparse matrix size is the same, why resource for 1152 cores are needed for 576 partitions on A, while only resource for 432 cores are needed for 144 partitions on A? If using 432 cores can solve the 144- partition Ax=b, why did it need 1152 cores to solve 576-partition Ax=b? I expected 576 cores could do the job that 432 cores did on the 576-partition Ax=b. This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c mpiexec -n 2 ./ex10 -f -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view ... then you'll see memory info provided by mumps. >> Good point. I will link in KSP to test it. * I suspect one rank was trying to collect all data from other ranks. Yes, analysis is sequential, and rhs vector must be in the host :-( In general, direct solvers cannot be scaled to very large num of cores. >> I meant rank 0 is trying to collect all NNZ from all other ranks. What other addition setting is needed for mumps such that it could deal with medium and large size problems? Run your code with option '-help |grep mumps' and experiment with various options, e.g., matrix orderings, nonzero-fills etc. You may also try superlu_dist. Good luck! >> superlu_dist failed in providing the solution for small size problems already. Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 14:44:54 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 14:44:54 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 2:36 PM, Jinquan Zhong wrote: > *>> That is a good point. In these applications, we don?t usually have a > good-conditioned matrix . The condition number is always around > 10^10--10^12. This is out of our control.* > This is something you may have to "take control" of if you want accurate solutions. You can also compute in quad precision (--with-precision=__float128), but AFAIK, there are no quad-precision distributed memory sparse direct solvers. > ** ** > > ** ** > > Is this the reason you want to use a direct solver instead of iterative > one?**** > > What do you mean "31k with a dense matrix included in the sparse matrix"?* > *** > > ** ** > > ** ** > > *>> We have a dense matrix embedded inside a sparse matrix. This dense > matrix usually accounts for 99% of the total nnz?s.* > What does that dense coupling represent? > ** > > ** ** > > How sparse is your matrix, e.g., nnz(A)/(m*m)=?**** > > ** ** > > *>> ~=0.4% for medium size problem and 0.04% for large size problem.* > Uhh, your numbers for the large problem are 178k^2 / 640k^2 = 7.8%. > Both a and b seem indicate that, you can use small num of cores to > generate original matrix A, but need more cores (resource) to solve A x =b. > > ** > > ** ** > > *>> My confusion is that since the sparse matrix size is the same, why > resource for 1152 cores are needed for 576 partitions on A, while only > resource for 432 cores are needed for 144 partitions on A? If using 432 > cores can solve the 144- partition Ax=b, why did it need 1152 cores to > solve 576-partition Ax=b? I expected 576 cores could do the job that 432 > cores did on the 576-partition Ax=b.* > The amount of fill depends on the size of minimal vertex separators. Sparse matrices with the same number of nonzeros and same number of nonzeros per row can have vertex separators that are orders of magnitude different in size. The fill is quadratic in the size of the separators and computation is cubic. Is your problem symmetric? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 14:48:46 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 19:48:46 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, October 23, 2012 12:45 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 2:36 PM, Jinquan Zhong > wrote: >> That is a good point. In these applications, we don?t usually have a good-conditioned matrix . The condition number is always around 10^10--10^12. This is out of our control. This is something you may have to "take control" of if you want accurate solutions. You can also compute in quad precision (--with-precision=__float128), but AFAIK, there are no quad-precision distributed memory sparse direct solvers. Is this the reason you want to use a direct solver instead of iterative one? What do you mean "31k with a dense matrix included in the sparse matrix"? >> We have a dense matrix embedded inside a sparse matrix. This dense matrix usually accounts for 99% of the total nnz?s. What does that dense coupling represent? >> Soil-structure interaction. How sparse is your matrix, e.g., nnz(A)/(m*m)=? >> ~=0.4% for medium size problem and 0.04% for large size problem. Uhh, your numbers for the large problem are 178k^2 / 640k^2 = 7.8%. >> my bad. Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b. >> My confusion is that since the sparse matrix size is the same, why resource for 1152 cores are needed for 576 partitions on A, while only resource for 432 cores are needed for 144 partitions on A? If using 432 cores can solve the 144- partition Ax=b, why did it need 1152 cores to solve 576-partition Ax=b? I expected 576 cores could do the job that 432 cores did on the 576-partition Ax=b. The amount of fill depends on the size of minimal vertex separators. Sparse matrices with the same number of nonzeros and same number of nonzeros per row can have vertex separators that are orders of magnitude different in size. The fill is quadratic in the size of the separators and computation is cubic. >> Could you be more specific? I am not quite with you yet. Is your problem symmetric? >> Yes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 16:14:01 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 16:14:01 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 2:48 PM, Jinquan Zhong wrote: > >> Soil-structure interaction. > Why is it dense? Is it effectively the solution of another equation? An integral operator? > > > ** > > The amount of fill depends on the size of minimal vertex separators. > Sparse matrices with the same number of nonzeros and same number of > nonzeros per row can have vertex separators that are orders of magnitude > different in size. The fill is quadratic in the size of the separators and > computation is cubic.**** > > >> Could you be more specific? I am not quite with you yet. > A 10x10x10000 3D problem with hex elements has about 1M dofs and about 27 nonzeros per row. The minimal vertex separator consists of 10^2 vertices, so the final dense matrix is 100x100. The direct solver is extremely fast for this problem. A 100x100x100 3D problem has the same number of dofs and nonzeros per row. The minimal vertex separator is 100^2 vertices, so the final dense matrix is 10000x10000 (and there are many pretty big dense matrices to get there). This problem requires on the order of 10000 times as much memory and 1000000 times as many flops. If we switch from a FEM discretization to a FD discretization with a stencil width of 3, the vertex separator grows by a factor of 3, increasing memory usage by a factor of 9 and flops by a factor of 27. If you replace that high order system with high order continuous Galerkin FEM using larger elements so the number of dofs is constant, the number of nonzeros in the matrix may grow, but the vertex separators go back to being the same as the original problem. > **** > > ** ** > > Is your problem symmetric?**** > > ** ** > > >> Yes. > Hong, is the Clique code in petsc-dev ready to use? There is no reason we should keep spending time dealing with MUMPS quirks and scalability problems on symmetric problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 17:15:39 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 22:15:39 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> Hong and Jed, For KSPSolver, what kind of PC is the most proper ? I tested ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr); It worked for small complex double matrix but not for the big ones. Here is my set up KSP ksp; /* linear solver context */ PC pc; ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr); KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSolve(ksp,b,*x);CHKERRQ(ierr); Do you see any problem? Thanks, Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Tuesday, October 23, 2012 12:03 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc Jinquan: I have a question on how to use mumps properly in PETSc. It appears that I didn't set up mumps right. I followed the example in http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html This example is for our internal testing, not intended for production runs. Suggest using PETSc high level KSP solver which provides more flexibility. to set up my program. Here is my situation on using the default setting PetscInt icntl_7 = 5; MatMumpsSetIcntl(F,7,icntl_7); in the example ex125.c: Using KSP sover, the mumps options can be chosen at runtime, e.g. ~petsc/src/ksp/ksp/examples/tutorials/ex2.c: mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_7 5 Norm of error 1.49777e-15 iterations 1 1. The program work fine on all small models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3. ^^^^ With a direct solver, residual = 10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ? Is this the reason you want to use a direct solver instead of iterative one? What do you mean "31k with a dense matrix included in the sparse matrix"? How sparse is your matrix, e.g., nnz(A)/(m*m)=? 2. The program has some issues on medium size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB. a. We used another software to generate sparse matrix by using 144 cores: i. When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. What is "another software "? b. We used another software to generate the same sparse matrix by using 576 cores: i. When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b. This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c mpiexec -n 2 ./ex10 -f -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view ... then you'll see memory info provided by mumps. 3. The program could not solve the large size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix). The full sparse matrix is sized at 511GB. a. We used another software to generate sparse matrix by using 900 cores: i. When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT provide the solution. Your computer system and software have limits. Find the answers to your 'medium size' problems first. My confusion starts from the medium size problem: * It seems something was not right in the default setting in ex125.c for these problems. * I got the info that METIS was used instead of ParMETIS in solving these problems. By default, petsc-mumps interface uses sequential symbolic factorization (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel. I tested it, but seems parmetis is still not used. Check mumps manual or contact mumps developer on how to use parmetis. * Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem. * I suspect one rank was trying to collect all data from other ranks. Yes, analysis is sequential, and rhs vector must be in the host :-( In general, direct solvers cannot be scaled to very large num of cores. What other addition setting is needed for mumps such that it could deal with medium and large size problems? Run your code with option '-help |grep mumps' and experiment with various options, e.g., matrix orderings, nonzero-fills etc. You may also try superlu_dist. Good luck! Do you guys have similar experience on that? I personally never used mumps or superlu_dist for such large matrices. Consult mumps developers. Hong Thanks, Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 23 17:19:10 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Oct 2012 18:19:10 -0400 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 6:15 PM, Jinquan Zhong wrote: > Hong and Jed,**** > > ** ** > > For KSPSolver, what kind of PC is the most proper ? I tested **** > > ** ** > > ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr);**** > > ** ** > > It worked for small complex double matrix but not for the big ones. Here > is my set up > Did you read the documentation on this PC? At all? That it solves the entire system on each proces? Matt > > > ** ** > > KSP ksp; /* linear solver context */**** > > PC pc;**** > > ** ** > > ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);**** > > ierr = > KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);**** > > **** > > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);**** > > ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr);**** > > > KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT);**** > > **** > > ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);**** > > ierr = KSPSolve(ksp,b,*x);CHKERRQ(ierr);**** > > ** ** > > Do you see any problem?**** > > ** ** > > Thanks,**** > > ** ** > > Jinquan**** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Hong Zhang > *Sent:* Tuesday, October 23, 2012 12:03 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > ** ** > > Jinquan:**** > > ** ** > > I have a question on how to use mumps properly in PETSc. It appears that > I didn?t set up mumps right. I followed the example in **** > > http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html > **** > > **** > > This example is for our internal testing, not intended for production runs. > **** > > Suggest using PETSc high level KSP solver which provides more flexibility. > **** > > **** > > to set up my program. Here is my situation on using the default setting * > *** > > **** > > PetscInt icntl_7 = 5;**** > > MatMumpsSetIcntl > (F,7,icntl_7);**** > > **** > > in the example ex125.c:**** > > Using KSP sover, the mumps options can be chosen at runtime,**** > > e.g. **** > > ~petsc/src/ksp/ksp/examples/tutorials/ex2.c:**** > > mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps > -mat_mumps_icntl_7 5**** > > Norm of error 1.49777e-15 iterations 1**** > > 1. The program work fine on all *small* models (sparse matrices > at the order of m= 894, 1097, 31k with a dense matrix included in the > sparse matrix). The residuals are at the magnitude of 10^-3.**** > > ^^^^**** > > With a direct solver, residual = 10^-3 indicates your matrix might be > very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ? > **** > > Is this the reason you want to use a direct solver instead of iterative > one?**** > > What do you mean "31k with a dense matrix included in the sparse matrix"?* > *** > > How sparse is your matrix, e.g., nnz(A)/(m*m)=?**** > > 2. The program has some issues on *medium* size problem (m=460k > with a dense matrix at the order of n=30k included in the sparse matrix). > The full sparse matrix is sized at 17GB.**** > > a. We used another software to generate sparse matrix by using 144 > cores: **** > > i. When > I used the resource from 144 cores (12 nodes with 48GB/node), it could not > provide the solution. There was a complain on the memory violation.**** > > ii. When > I used the resource from 432 cores (36 nodes with 48GB/node), it provided > the solution. **** > > Direct solvers are notoriously memory consuming. It seems your matrix is > quite dense, requiring more memory than 144 cores could provide. **** > > What is "another software "?**** > > b. We used another software to generate the same sparse matrix by > using 576 cores: **** > > i. When > I used the resource from 576 cores (48 nodes with 48GB/node), it could not > provide the solution. There was a complain on the memory violation.**** > > ii. When > I used the resource from 1152 cores (96 nodes with 48GB/node), it provided > the solution. **** > > Both a and b seem indicate that, you can use small num of cores to > generate original matrix A, but need more cores (resource) to solve A x =b. > **** > > This is because A = LU, the factored matrices L and U require far more > memory than original A. Run your code using KSP with your matrix data and > option -ksp_view**** > > e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c**** > > mpiexec -n 2 ./ex10 -f -pc_type lu > -pc_factor_mat_solver_package mumps -ksp_view**** > > ...**** > > then you'll see memory info provided by mumps. **** > > 3. The program could not solve the *large* size problem (m=640k > with a dense matrix at the order of n=178k included in the sparse matrix). > The full sparse matrix is sized at 511GB.**** > > a. We used another software to generate sparse matrix by using 900 > cores: **** > > i. When > I used the resource from 900 cores (75 nodes with 48GB/node), it could not > provide the solution. There was a complain on the memory violation.**** > > ii. When > I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL > COULD NOT provide the solution. **** > > Your computer system and software have limits. Find the answers to your > 'medium size' problems first. **** > > ** ** > > My confusion starts from the medium size problem:**** > > ? It seems something was not right in the default setting in > ex125.c for these problems. **** > > ? I got the info that METIS was used instead of ParMETIS in > solving these problems. **** > > By default, petsc-mumps interface uses sequential symbolic factorization > (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel.**** > > I tested it, but seems parmetis is still not used. Check mumps manual **** > > or contact mumps developer on how to use parmetis.**** > > ? Furthermore, it appears that there was unreasonable demand on > the solver even on the medium size problem. **** > > ? I suspect one rank was trying to collect all data from other > ranks. **** > > Yes, analysis is sequential, and rhs vector must be in the host :-( **** > > In general, direct solvers cannot be scaled to very large num of cores.*** > * > > **** > > What other addition setting is needed for mumps such that it could deal > with medium and large size problems?**** > > **** > > Run your code with option '-help |grep mumps' and experiment with**** > > various options, e.g., matrix orderings, nonzero-fills etc. **** > > You may also try superlu_dist. Good luck!**** > > **** > > Do you guys have similar experience on that?**** > > I personally never used mumps or superlu_dist for such large matrices.** > ** > > Consult mumps developers.**** > > ** ** > > Hong**** > > **** > > Thanks,**** > > **** > > Jinquan**** > > **** > > ** ** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 17:21:03 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 22:21:03 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> That is new for me. What would you suggest, Matt? Thanks, Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, October 23, 2012 3:19 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 6:15 PM, Jinquan Zhong > wrote: Hong and Jed, For KSPSolver, what kind of PC is the most proper ? I tested ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr); It worked for small complex double matrix but not for the big ones. Here is my set up Did you read the documentation on this PC? At all? That it solves the entire system on each proces? Matt KSP ksp; /* linear solver context */ PC pc; ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr); KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSolve(ksp,b,*x);CHKERRQ(ierr); Do you see any problem? Thanks, Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Hong Zhang Sent: Tuesday, October 23, 2012 12:03 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc Jinquan: I have a question on how to use mumps properly in PETSc. It appears that I didn't set up mumps right. I followed the example in http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html This example is for our internal testing, not intended for production runs. Suggest using PETSc high level KSP solver which provides more flexibility. to set up my program. Here is my situation on using the default setting PetscInt icntl_7 = 5; MatMumpsSetIcntl(F,7,icntl_7); in the example ex125.c: Using KSP sover, the mumps options can be chosen at runtime, e.g. ~petsc/src/ksp/ksp/examples/tutorials/ex2.c: mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_7 5 Norm of error 1.49777e-15 iterations 1 1. The program work fine on all small models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3. ^^^^ With a direct solver, residual = 10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ? Is this the reason you want to use a direct solver instead of iterative one? What do you mean "31k with a dense matrix included in the sparse matrix"? How sparse is your matrix, e.g., nnz(A)/(m*m)=? 2. The program has some issues on medium size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB. a. We used another software to generate sparse matrix by using 144 cores: i. When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. What is "another software "? b. We used another software to generate the same sparse matrix by using 576 cores: i. When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b. This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c mpiexec -n 2 ./ex10 -f -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view ... then you'll see memory info provided by mumps. 3. The program could not solve the large size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix). The full sparse matrix is sized at 511GB. a. We used another software to generate sparse matrix by using 900 cores: i. When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation. ii. When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT provide the solution. Your computer system and software have limits. Find the answers to your 'medium size' problems first. My confusion starts from the medium size problem: * It seems something was not right in the default setting in ex125.c for these problems. * I got the info that METIS was used instead of ParMETIS in solving these problems. By default, petsc-mumps interface uses sequential symbolic factorization (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel. I tested it, but seems parmetis is still not used. Check mumps manual or contact mumps developer on how to use parmetis. * Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem. * I suspect one rank was trying to collect all data from other ranks. Yes, analysis is sequential, and rhs vector must be in the host :-( In general, direct solvers cannot be scaled to very large num of cores. What other addition setting is needed for mumps such that it could deal with medium and large size problems? Run your code with option '-help |grep mumps' and experiment with various options, e.g., matrix orderings, nonzero-fills etc. You may also try superlu_dist. Good luck! Do you guys have similar experience on that? I personally never used mumps or superlu_dist for such large matrices. Consult mumps developers. Hong Thanks, Jinquan -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 17:28:29 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 17:28:29 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong wrote: > That is new for me. What would you suggest, Matt? Were you using LU or Cholesky before? That is the difference between -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps to choose MUMPS. You can access the MUMPS options with -mat_mumps_icntl_opaquenumber. It looks like PETSc's Clique interface does not work in parallel. (A student was working on it recently and it seems to work in serial.) When it is fixed to work in parallel, that is almost certainly what you should use. Alternatively, there may well be a Fast method, depending on the structure of the system and that fat dense block. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 18:17:55 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 23:17:55 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> Thanks, Jed. Any way to get the condition number. I used -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 It didn?t work. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, October 23, 2012 3:28 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote: That is new for me. What would you suggest, Matt? Were you using LU or Cholesky before? That is the difference between -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps to choose MUMPS. You can access the MUMPS options with -mat_mumps_icntl_opaquenumber. It looks like PETSc's Clique interface does not work in parallel. (A student was working on it recently and it seems to work in serial.) When it is fixed to work in parallel, that is almost certainly what you should use. Alternatively, there may well be a Fast method, depending on the structure of the system and that fat dense block. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 23 18:30:29 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Oct 2012 19:30:29 -0400 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 7:17 PM, Jinquan Zhong wrote: > Thanks, Jed.**** > > ** ** > > Any way to get the condition number. I used **** > > ** ** > > -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000**** > > ** ** > > It didn?t work. > This is not helpful. Would you understand this if someone mailed you "It didn't work"? Send the output. Matt > > > Jinquan**** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Tuesday, October 23, 2012 3:28 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > ** ** > > On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote:**** > > That is new for me. What would you suggest, Matt?**** > > ** ** > > Were you using LU or Cholesky before? That is the difference between > -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps > to choose MUMPS. You can access the MUMPS options with > -mat_mumps_icntl_opaquenumber.**** > > ** ** > > It looks like PETSc's Clique interface does not work in parallel. (A > student was working on it recently and it seems to work in serial.) When it > is fixed to work in parallel, that is almost certainly what you should use. > Alternatively, there may well be a Fast method, depending on the structure > of the system and that fat dense block.**** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 18:33:37 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 18:33:37 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 6:17 PM, Jinquan Zhong wrote: > Any way to get the condition number. I used **** > > ** ** > > -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000**** > > ** ** > > It didn?t work. > First, as a matter of course, remove "It didn't work" from your vocabulary. Please never utter those words on any mailing list ever again. It wastes everyone's time. It is estimating the singular values of the *preconditioned* operator. Use -pc_type none to remove the preconditioner. Note that this is generally extremely inaccurate for the smallest eigenvalue, especially for an ill-conditioned problem and when you don't run to convergence. It can over-estimate the smallest singular value by many orders of magnitude. In general, it is difficult to estimate the smallest singular value without a * solver*. You can use SLEPc to accurately and efficiently compute the smallest singular values, provided you have a reasonable preconditioner (or direct solver) for the system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 18:34:40 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 23:34:40 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE40D@exchange.scsolutions.com> That is true. I used -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_4 2 -mat_mumps_icntl_5 0 -mat_mumps_icntl_18 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 Here is the info I got from the setting I had. I didn't see the condition number appeared: Entering ZMUMPS driver with JOB, N, NZ = 1 894 0 ZMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** ** Max-trans not allowed because matrix is distributed Using ParMETIS for parallel ordering. Structual symmetry is:100% WARNING: Largest root node of size 173 not selected for parallel execution Leaving analysis phase with ... INFOG(1) = 0 INFOG(2) = 0 -- (20) Number of entries in factors (estim.) = 306174 -- (3) Storage of factors (REAL, estimated) = 306174 -- (4) Storage of factors (INT , estimated) = 7102 -- (5) Maximum frontal size (estimated) = 495 -- (6) Number of nodes in the tree = 66 -- (32) Type of analysis effectively used = 2 -- (7) Ordering option effectively used = 2 ICNTL(6) Maximum transversal option = 0 ICNTL(7) Pivot order option = 7 Percentage of memory relaxation (effective) = 25 Number of level 2 nodes = 0 Number of split nodes = 0 RINFOG(1) Operations during elimination (estim)= 8.929D+07 Distributed matrix entry format (ICNTL(18)) = 3 ** Rank of proc needing largest memory in IC facto : 1 ** Estimated corresponding MBYTES for IC facto : 43 ** Estimated avg. MBYTES per work. proc at facto (IC) : 39 ** TOTAL space in MBYTES for IC factorization : 156 ** Rank of proc needing largest memory for OOC facto : 1 ** Estimated corresponding MBYTES for OOC facto : 46 ** Estimated avg. MBYTES per work. proc at facto (OOC) : 41 ** TOTAL space in MBYTES for OOC factorization : 165 Entering ZMUMPS driver with JOB, N, NZ = 2 894 263360 ****** FACTORIZATION STEP ******** GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... NUMBER OF WORKING PROCESSES = 4 OUT-OF-CORE OPTION (ICNTL(22)) = 0 REAL SPACE FOR FACTORS = 306174 INTEGER SPACE FOR FACTORS = 7102 MAXIMUM FRONTAL SIZE (ESTIMATED) = 495 NUMBER OF NODES IN THE TREE = 66 Convergence error after scaling for ONE-NORM (option 7/8) = 0.31D+00 Maximum effective relaxed size of S = 306300 Average effective relaxed size of S = 185353 REDISTRIB: TOTAL DATA LOCAL/SENT = 68276 195084 GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.0145 ** Memory relaxation parameter ( ICNTL(14) ) : 25 ** Rank of processor needing largest memory in facto : 1 ** Space in MBYTES used by this processor for facto : 43 ** Avg. Space in MBYTES per working proc during facto : 39 ELAPSED TIME FOR FACTORIZATION = 0.0862 Maximum effective space used in S (KEEP8(67) = 245025 Average effective space used in S (KEEP8(67) = 119077 ** EFF Min: Rank of processor needing largest memory : 1 ** EFF Min: Space in MBYTES used by this processor : 42 ** EFF Min: Avg. Space in MBYTES per working proc : 37 GLOBAL STATISTICS RINFOG(2) OPERATIONS IN NODE ASSEMBLY = 2.372D+05 ------(3) OPERATIONS IN NODE ELIMINATION= 8.929D+07 INFOG (9) REAL SPACE FOR FACTORS = 306184 INFOG(10) INTEGER SPACE FOR FACTORS = 7104 INFOG(11) MAXIMUM FRONT SIZE = 495 INFOG(29) NUMBER OF ENTRIES IN FACTORS = 306184 INFOG(12) NB OF OFF DIAGONAL PIVOTS = 29 INFOG(13) NUMBER OF DELAYED PIVOTS = 1 INFOG(14) NUMBER OF MEMORY COMPRESS = 0 KEEP8(108) Extra copies IP stacking = 0 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 0 KSP Residual norm 1.890433086271e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 1 KSP Residual norm 1.804170434909e-06 % max 1.000000180789e+00 min 1.000000180789e+00 max/min 1.000000000000e+00 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 2 KSP Residual norm 4.466758995607e-13 % max 1.000000467998e+00 min 9.999992798573e-01 max/min 1.000001188141e+00 KSP Object: 4 MPI processes type: gmres GMRES: restart=1000, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-12, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 4 MPI processes type: mpiaij rows=894, cols=894 package used to perform factorization: mumps total: nonzeros=306174, allocated nonzeros=306174 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 6 ICNTL(4) (level of printing): 2 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -8 ICNTL(28) (use parallel or sequential ordering): 2 ICNTL(29) (parallel ordering): 2 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0 CNTL(4) (value of static pivoting): -1 CNTL(5) (fixation for null pivots): 0 RINFO(1) (local estimated flops for the elimination after analysis): [0] 4.25638e+06 [1] 7.22129e+07 [2] 8.01308e+06 [3] 4.81122e+06 RINFO(2) (local estimated flops for the assembly after factorization): [0] 68060 [1] 5860 [2] 83570 [3] 79676 RINFO(3) (local estimated flops for the elimination after factorization): [0] 4.25638e+06 [1] 7.2213e+07 [2] 8.01308e+06 [3] 4.81122e+06 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 37 [1] 43 [2] 38 [3] 38 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 37 [1] 43 [2] 38 [3] 38 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 350 [1] 319 [2] 134 [3] 91 RINFOG(1) (global estimated flops for the elimination after analysis): 8.92936e+07 RINFOG(2) (global estimated flops for the assembly after factorization): 237166 RINFOG(3) (global estimated flops for the elimination after factorization): 8.92937e+07 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 306174 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 7102 INFOG(5) (estimated maximum front size in the complete tree): 495 INFOG(6) (number of nodes in the complete tree): 66 INFOG(7) (ordering option effectively use after analysis): 2 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 306184 INFOG(10) (total integer space store the matrix factors after factorization): 7104 INFOG(11) (order of largest frontal matrix after factorization): 495 INFOG(12) (number of off-diagonal pivots): 29 INFOG(13) (number of delayed pivots after factorization): 1 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 43 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 156 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 43 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 156 INFOG(20) (estimated number of entries in the factors): 306174 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 42 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 150 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=894, cols=894 total: nonzeros=263360, allocated nonzeros=263360 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 47 nodes, limit used is 5 >> # of iterations: 2 KSPConvergedReason: 3 Entering ZMUMPS driver with JOB, N, NZ = -2 894 263360 Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, October 23, 2012 4:30 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 7:17 PM, Jinquan Zhong > wrote: Thanks, Jed. Any way to get the condition number. I used -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 It didn't work. This is not helpful. Would you understand this if someone mailed you "It didn't work"? Send the output. Matt Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, October 23, 2012 3:28 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote: That is new for me. What would you suggest, Matt? Were you using LU or Cholesky before? That is the difference between -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps to choose MUMPS. You can access the MUMPS options with -mat_mumps_icntl_opaquenumber. It looks like PETSc's Clique interface does not work in parallel. (A student was working on it recently and it seems to work in serial.) When it is fixed to work in parallel, that is almost certainly what you should use. Alternatively, there may well be a Fast method, depending on the structure of the system and that fat dense block. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 23 18:38:21 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 23 Oct 2012 18:38:21 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE40D@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE40D@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 6:34 PM, Jinquan Zhong wrote: > * > That is true. I used* > > ** ** > > -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view > -mat_mumps_icntl_4 2 -mat_mumps_icntl_5 0 -mat_mumps_icntl_18 3 > -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_type gmres > -ksp_monitor_singular_value -ksp_gmres_restart 1000 > Read my reply about using GMRES to estimate condition number _while_ using a preconditioner. > **** > > ** ** > > ** ** > > *Here is the info I got from the setting I had. I didn?t see the > condition number appeared:* > > ** ** > > ** ** > > Entering ZMUMPS driver with JOB, N, NZ = 1 894 0*** > * > > ** ** > > ZMUMPS 4.10.0 **** > > L U Solver for unsymmetric matrices**** > > Type of parallelism: Working host**** > > ** ** > > ****** ANALYSIS STEP ************ > > ** ** > > ** Max-trans not allowed because matrix is distributed**** > > Using ParMETIS for parallel ordering.**** > > Structual symmetry is:100%**** > > WARNING: Largest root node of size 173 not selected for > parallel execution**** > > ** ** > > Leaving analysis phase with ...**** > > INFOG(1) = 0**** > > INFOG(2) = 0**** > > -- (20) Number of entries in factors (estim.) = 306174**** > > -- (3) Storage of factors (REAL, estimated) = 306174**** > > -- (4) Storage of factors (INT , estimated) = 7102**** > > -- (5) Maximum frontal size (estimated) = 495**** > > -- (6) Number of nodes in the tree = 66**** > > -- (32) Type of analysis effectively used = 2**** > > -- (7) Ordering option effectively used = 2**** > > ICNTL(6) Maximum transversal option = 0**** > > ICNTL(7) Pivot order option = 7**** > > Percentage of memory relaxation (effective) = 25**** > > Number of level 2 nodes = 0**** > > Number of split nodes = 0**** > > RINFOG(1) Operations during elimination (estim)= 8.929D+07**** > > Distributed matrix entry format (ICNTL(18)) = 3**** > > ** Rank of proc needing largest memory in IC facto : 1**** > > ** Estimated corresponding MBYTES for IC facto : 43**** > > ** Estimated avg. MBYTES per work. proc at facto (IC) : 39**** > > ** TOTAL space in MBYTES for IC factorization : 156**** > > ** Rank of proc needing largest memory for OOC facto : 1**** > > ** Estimated corresponding MBYTES for OOC facto : 46**** > > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 41**** > > ** TOTAL space in MBYTES for OOC factorization : 165**** > > Entering ZMUMPS driver with JOB, N, NZ = 2 894 263360*** > * > > ** ** > > ****** FACTORIZATION STEP ************ > > ** ** > > ** ** > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...**** > > NUMBER OF WORKING PROCESSES = 4**** > > OUT-OF-CORE OPTION (ICNTL(22)) = 0**** > > REAL SPACE FOR FACTORS = 306174**** > > INTEGER SPACE FOR FACTORS = 7102**** > > MAXIMUM FRONTAL SIZE (ESTIMATED) = 495**** > > NUMBER OF NODES IN THE TREE = 66**** > > Convergence error after scaling for ONE-NORM (option 7/8) = 0.31D+00**** > > Maximum effective relaxed size of S = 306300**** > > Average effective relaxed size of S = 185353**** > > ** ** > > REDISTRIB: TOTAL DATA LOCAL/SENT = 68276 195084**** > > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.0145**** > > ** Memory relaxation parameter ( ICNTL(14) ) : 25**** > > ** Rank of processor needing largest memory in facto : 1**** > > ** Space in MBYTES used by this processor for facto : 43**** > > ** Avg. Space in MBYTES per working proc during facto : 39**** > > ** ** > > ELAPSED TIME FOR FACTORIZATION = 0.0862**** > > Maximum effective space used in S (KEEP8(67) = 245025**** > > Average effective space used in S (KEEP8(67) = 119077**** > > ** EFF Min: Rank of processor needing largest memory : 1**** > > ** EFF Min: Space in MBYTES used by this processor : 42**** > > ** EFF Min: Avg. Space in MBYTES per working proc : 37**** > > ** ** > > GLOBAL STATISTICS **** > > RINFOG(2) OPERATIONS IN NODE ASSEMBLY = 2.372D+05**** > > ------(3) OPERATIONS IN NODE ELIMINATION= 8.929D+07**** > > INFOG (9) REAL SPACE FOR FACTORS = 306184**** > > INFOG(10) INTEGER SPACE FOR FACTORS = 7104**** > > INFOG(11) MAXIMUM FRONT SIZE = 495**** > > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 306184**** > > INFOG(12) NB OF OFF DIAGONAL PIVOTS = 29**** > > INFOG(13) NUMBER OF DELAYED PIVOTS = 1**** > > INFOG(14) NUMBER OF MEMORY COMPRESS = 0**** > > KEEP8(108) Extra copies IP stacking = 0**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > ** ** > > ** ** > > ****** SOLVE & CHECK STEP ************ > > ** ** > > ** ** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 0 KSP Residual norm 1.890433086271e+01 % max 1.000000000000e+00 min > 1.000000000000e+00 max/min 1.000000000000e+00**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > ** ** > > ** ** > > ****** SOLVE & CHECK STEP ************ > > ** ** > > ** ** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 1 KSP Residual norm 1.804170434909e-06 % max 1.000000180789e+00 min > 1.000000180789e+00 max/min 1.000000000000e+00**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > ** ** > > ** ** > > ****** SOLVE & CHECK STEP ************ > > ** ** > > ** ** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 2 KSP Residual norm 4.466758995607e-13 % max 1.000000467998e+00 min > 9.999992798573e-01 max/min 1.000001188141e+00 > Here are your estimates of max and min singular values, and their ratio (the condition number). These are for the *preconditioned operator*. Since you use a direct solver, it is almost exactly 1. > **** > > KSP Object: 4 MPI processes**** > > type: gmres**** > > GMRES: restart=1000, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement**** > > GMRES: happy breakdown tolerance 1e-30**** > > maximum iterations=10000, initial guess is zero**** > > tolerances: relative=1e-12, absolute=1e-12, divergence=10000**** > > left preconditioning**** > > using PRECONDITIONED norm type for convergence test**** > > PC Object: 4 MPI processes**** > > type: lu**** > > LU: out-of-place factorization**** > > tolerance for zero pivot 2.22045e-14**** > > matrix ordering: natural**** > > factor fill ratio given 0, needed 0**** > > Factored matrix follows:**** > > Matrix Object: 4 MPI processes**** > > type: mpiaij**** > > rows=894, cols=894**** > > package used to perform factorization: mumps**** > > total: nonzeros=306174, allocated nonzeros=306174**** > > total number of mallocs used during MatSetValues calls =0**** > > MUMPS run parameters:**** > > SYM (matrix type): 0 **** > > PAR (host participation): 1 **** > > ICNTL(1) (output for error): 6 **** > > ICNTL(2) (output of diagnostic msg): 0 **** > > ICNTL(3) (output for global info): 6 **** > > ICNTL(4) (level of printing): 2 **** > > ICNTL(5) (input mat struct): 0 **** > > ICNTL(6) (matrix prescaling): 7 **** > > ICNTL(7) (sequentia matrix ordering):7 **** > > ICNTL(8) (scalling strategy): 77 **** > > ICNTL(10) (max num of refinements): 0 **** > > ICNTL(11) (error analysis): 0 **** > > ICNTL(12) (efficiency control): 1 ** > ** > > ICNTL(13) (efficiency control): 0 ** > ** > > ICNTL(14) (percentage of estimated workspace increase): 20 * > *** > > ICNTL(18) (input mat struct): 3 ** > ** > > ICNTL(19) (Shur complement info): 0 ** > ** > > ICNTL(20) (rhs sparse pattern): 0 ** > ** > > ICNTL(21) (solution struct): 1 ** > ** > > ICNTL(22) (in-core/out-of-core facility): 0 ** > ** > > ICNTL(23) (max size of memory can be allocated locally):0 ** > ** > > ICNTL(24) (detection of null pivot rows): 0 ** > ** > > ICNTL(25) (computation of a null space basis): 0 ** > ** > > ICNTL(26) (Schur options for rhs or solution): 0 ** > ** > > ICNTL(27) (experimental parameter): -8 * > *** > > ICNTL(28) (use parallel or sequential ordering): 2 ** > ** > > ICNTL(29) (parallel ordering): 2 ** > ** > > ICNTL(30) (user-specified set of entries in inv(A)): 0 ** > ** > > ICNTL(31) (factors is discarded in the solve phase): 0 ** > ** > > ICNTL(33) (compute determinant): 0 ** > ** > > CNTL(1) (relative pivoting threshold): 0.01 **** > > CNTL(2) (stopping criterion of refinement): 1.49012e-08 **** > > CNTL(3) (absolute pivoting threshold): 0 **** > > CNTL(4) (value of static pivoting): -1 **** > > CNTL(5) (fixation for null pivots): 0 **** > > RINFO(1) (local estimated flops for the elimination after > analysis): **** > > [0] 4.25638e+06 **** > > [1] 7.22129e+07 **** > > [2] 8.01308e+06 **** > > [3] 4.81122e+06 **** > > RINFO(2) (local estimated flops for the assembly after > factorization): **** > > [0] 68060 **** > > [1] 5860 **** > > [2] 83570 **** > > [3] 79676 **** > > RINFO(3) (local estimated flops for the elimination after > factorization): **** > > [0] 4.25638e+06 **** > > [1] 7.2213e+07 **** > > [2] 8.01308e+06 **** > > [3] 4.81122e+06 **** > > INFO(15) (estimated size of (in MB) MUMPS internal data for > running numerical factorization): **** > > [0] 37 **** > > [1] 43 **** > > [2] 38 **** > > [3] 38 **** > > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): **** > > [0] 37 **** > > [1] 43 **** > > [2] 38 **** > > [3] 38 **** > > INFO(23) (num of pivots eliminated on this processor after > factorization): **** > > [0] 350 **** > > [1] 319 **** > > [2] 134 **** > > [3] 91 **** > > RINFOG(1) (global estimated flops for the elimination after > analysis): 8.92936e+07 **** > > RINFOG(2) (global estimated flops for the assembly after > factorization): 237166 **** > > RINFOG(3) (global estimated flops for the elimination after > factorization): 8.92937e+07 **** > > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0,0)*(2^0)**** > > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 306174 **** > > INFOG(4) (estimated integer workspace for factors on all > processors after analysis): 7102 **** > > INFOG(5) (estimated maximum front size in the complete > tree): 495 **** > > INFOG(6) (number of nodes in the complete tree): 66 **** > > INFOG(7) (ordering option effectively use after analysis): 2 > **** > > INFOG(8) (structural symmetry in percent of the permuted > matrix after analysis): 100 **** > > INFOG(9) (total real/complex workspace to store the matrix > factors after factorization): 306184 **** > > INFOG(10) (total integer space store the matrix factors > after factorization): 7104 **** > > INFOG(11) (order of largest frontal matrix after > factorization): 495 **** > > INFOG(12) (number of off-diagonal pivots): 29 **** > > INFOG(13) (number of delayed pivots after factorization): 1 > **** > > INFOG(14) (number of memory compress after factorization): 0 > **** > > INFOG(15) (number of steps of iterative refinement after > solution): 0 **** > > INFOG(16) (estimated size (in MB) of all MUMPS internal data > for factorization after analysis: value on the most memory consuming > processor): 43 **** > > INFOG(17) (estimated size of all MUMPS internal data for > factorization after analysis: sum over all processors): 156 **** > > INFOG(18) (size of all MUMPS internal data allocated during > factorization: value on the most memory consuming processor): 43 **** > > INFOG(19) (size of all MUMPS internal data allocated during > factorization: sum over all processors): 156 **** > > INFOG(20) (estimated number of entries in the factors): > 306174 **** > > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 42 **** > > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 150 **** > > INFOG(23) (after analysis: value of ICNTL(6) effectively > used): 0 **** > > INFOG(24) (after analysis: value of ICNTL(12) effectively > used): 1 **** > > INFOG(25) (after factorization: number of pivots modified by > static pivoting): 0 **** > > linear system matrix = precond matrix:**** > > Matrix Object: 4 MPI processes**** > > type: mpiaij**** > > rows=894, cols=894**** > > total: nonzeros=263360, allocated nonzeros=263360**** > > total number of mallocs used during MatSetValues calls =0**** > > using I-node (on process 0) routines: found 47 nodes, limit used is 5 > **** > > >> # of iterations: 2**** > > KSPConvergedReason: 3**** > > Entering ZMUMPS driver with JOB, N, NZ = -2 894 263360*** > * > > ** ** > > ** ** > > Jinquan**** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Tuesday, October 23, 2012 4:30 PM > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > ** ** > > On Tue, Oct 23, 2012 at 7:17 PM, Jinquan Zhong > wrote:**** > > Thanks, Jed.**** > > **** > > Any way to get the condition number. I used **** > > **** > > -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000**** > > **** > > It didn?t work.**** > > ** ** > > This is not helpful. Would you understand this if someone mailed you "It > didn't work"? Send the output.**** > > ** ** > > Matt**** > > **** > > **** > > Jinquan**** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Tuesday, October 23, 2012 3:28 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > **** > > On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote:**** > > That is new for me. What would you suggest, Matt?**** > > **** > > Were you using LU or Cholesky before? That is the difference between > -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps > to choose MUMPS. You can access the MUMPS options with > -mat_mumps_icntl_opaquenumber.**** > > **** > > It looks like PETSc's Clique interface does not work in parallel. (A > student was working on it recently and it seems to work in serial.) When it > is fixed to work in parallel, that is almost certainly what you should use. > Alternatively, there may well be a Fast method, depending on the structure > of the system and that fat dense block.**** > > > > **** > > ** ** > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Tue Oct 23 18:42:12 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Tue, 23 Oct 2012 23:42:12 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE40D@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704EE41F@exchange.scsolutions.com> I am seeing 652 KSP Residual norm 6.568962571627e+02 % max 1.464924876151e+14 min 8.431983181754e+03 max/min 1.737343214015e+10 653 KSP Residual norm 6.562904211497e+02 % max 1.464924876151e+14 min 8.431776971754e+03 max/min 1.737385702988e+10 when I used -pc_type none Does that indicate the Cond# is 1.7x10e10? Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, October 23, 2012 4:38 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 6:34 PM, Jinquan Zhong > wrote: That is true. I used -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_4 2 -mat_mumps_icntl_5 0 -mat_mumps_icntl_18 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 Read my reply about using GMRES to estimate condition number _while_ using a preconditioner. Here is the info I got from the setting I had. I didn?t see the condition number appeared: Entering ZMUMPS driver with JOB, N, NZ = 1 894 0 ZMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** ** Max-trans not allowed because matrix is distributed Using ParMETIS for parallel ordering. Structual symmetry is:100% WARNING: Largest root node of size 173 not selected for parallel execution Leaving analysis phase with ... INFOG(1) = 0 INFOG(2) = 0 -- (20) Number of entries in factors (estim.) = 306174 -- (3) Storage of factors (REAL, estimated) = 306174 -- (4) Storage of factors (INT , estimated) = 7102 -- (5) Maximum frontal size (estimated) = 495 -- (6) Number of nodes in the tree = 66 -- (32) Type of analysis effectively used = 2 -- (7) Ordering option effectively used = 2 ICNTL(6) Maximum transversal option = 0 ICNTL(7) Pivot order option = 7 Percentage of memory relaxation (effective) = 25 Number of level 2 nodes = 0 Number of split nodes = 0 RINFOG(1) Operations during elimination (estim)= 8.929D+07 Distributed matrix entry format (ICNTL(18)) = 3 ** Rank of proc needing largest memory in IC facto : 1 ** Estimated corresponding MBYTES for IC facto : 43 ** Estimated avg. MBYTES per work. proc at facto (IC) : 39 ** TOTAL space in MBYTES for IC factorization : 156 ** Rank of proc needing largest memory for OOC facto : 1 ** Estimated corresponding MBYTES for OOC facto : 46 ** Estimated avg. MBYTES per work. proc at facto (OOC) : 41 ** TOTAL space in MBYTES for OOC factorization : 165 Entering ZMUMPS driver with JOB, N, NZ = 2 894 263360 ****** FACTORIZATION STEP ******** GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ... NUMBER OF WORKING PROCESSES = 4 OUT-OF-CORE OPTION (ICNTL(22)) = 0 REAL SPACE FOR FACTORS = 306174 INTEGER SPACE FOR FACTORS = 7102 MAXIMUM FRONTAL SIZE (ESTIMATED) = 495 NUMBER OF NODES IN THE TREE = 66 Convergence error after scaling for ONE-NORM (option 7/8) = 0.31D+00 Maximum effective relaxed size of S = 306300 Average effective relaxed size of S = 185353 REDISTRIB: TOTAL DATA LOCAL/SENT = 68276 195084 GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.0145 ** Memory relaxation parameter ( ICNTL(14) ) : 25 ** Rank of processor needing largest memory in facto : 1 ** Space in MBYTES used by this processor for facto : 43 ** Avg. Space in MBYTES per working proc during facto : 39 ELAPSED TIME FOR FACTORIZATION = 0.0862 Maximum effective space used in S (KEEP8(67) = 245025 Average effective space used in S (KEEP8(67) = 119077 ** EFF Min: Rank of processor needing largest memory : 1 ** EFF Min: Space in MBYTES used by this processor : 42 ** EFF Min: Avg. Space in MBYTES per working proc : 37 GLOBAL STATISTICS RINFOG(2) OPERATIONS IN NODE ASSEMBLY = 2.372D+05 ------(3) OPERATIONS IN NODE ELIMINATION= 8.929D+07 INFOG (9) REAL SPACE FOR FACTORS = 306184 INFOG(10) INTEGER SPACE FOR FACTORS = 7104 INFOG(11) MAXIMUM FRONT SIZE = 495 INFOG(29) NUMBER OF ENTRIES IN FACTORS = 306184 INFOG(12) NB OF OFF DIAGONAL PIVOTS = 29 INFOG(13) NUMBER OF DELAYED PIVOTS = 1 INFOG(14) NUMBER OF MEMORY COMPRESS = 0 KEEP8(108) Extra copies IP stacking = 0 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 0 KSP Residual norm 1.890433086271e+01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 1 KSP Residual norm 1.804170434909e-06 % max 1.000000180789e+00 min 1.000000180789e+00 max/min 1.000000000000e+00 Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360 ****** SOLVE & CHECK STEP ******** STATISTICS PRIOR SOLVE PHASE ........... NUMBER OF RIGHT-HAND-SIDES = 1 BLOCKING FACTOR FOR MULTIPLE RHS = 1 ICNTL (9) = 1 --- (10) = 0 --- (11) = 0 --- (20) = 0 --- (21) = 1 --- (30) = 0 ** Rank of processor needing largest memory in solve : 1 ** Space in MBYTES used by this processor for solve : 6 ** Avg. Space in MBYTES per working proc during solve : 3 2 KSP Residual norm 4.466758995607e-13 % max 1.000000467998e+00 min 9.999992798573e-01 max/min 1.000001188141e+00 Here are your estimates of max and min singular values, and their ratio (the condition number). These are for the preconditioned operator. Since you use a direct solver, it is almost exactly 1. KSP Object: 4 MPI processes type: gmres GMRES: restart=1000, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-12, absolute=1e-12, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0, needed 0 Factored matrix follows: Matrix Object: 4 MPI processes type: mpiaij rows=894, cols=894 package used to perform factorization: mumps total: nonzeros=306174, allocated nonzeros=306174 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 6 ICNTL(4) (level of printing): 2 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequentia matrix ordering):7 ICNTL(8) (scalling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 3 ICNTL(19) (Shur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 1 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -8 ICNTL(28) (use parallel or sequential ordering): 2 ICNTL(29) (parallel ordering): 2 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0 CNTL(4) (value of static pivoting): -1 CNTL(5) (fixation for null pivots): 0 RINFO(1) (local estimated flops for the elimination after analysis): [0] 4.25638e+06 [1] 7.22129e+07 [2] 8.01308e+06 [3] 4.81122e+06 RINFO(2) (local estimated flops for the assembly after factorization): [0] 68060 [1] 5860 [2] 83570 [3] 79676 RINFO(3) (local estimated flops for the elimination after factorization): [0] 4.25638e+06 [1] 7.2213e+07 [2] 8.01308e+06 [3] 4.81122e+06 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 37 [1] 43 [2] 38 [3] 38 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 37 [1] 43 [2] 38 [3] 38 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 350 [1] 319 [2] 134 [3] 91 RINFOG(1) (global estimated flops for the elimination after analysis): 8.92936e+07 RINFOG(2) (global estimated flops for the assembly after factorization): 237166 RINFOG(3) (global estimated flops for the elimination after factorization): 8.92937e+07 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 306174 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 7102 INFOG(5) (estimated maximum front size in the complete tree): 495 INFOG(6) (number of nodes in the complete tree): 66 INFOG(7) (ordering option effectively use after analysis): 2 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 306184 INFOG(10) (total integer space store the matrix factors after factorization): 7104 INFOG(11) (order of largest frontal matrix after factorization): 495 INFOG(12) (number of off-diagonal pivots): 29 INFOG(13) (number of delayed pivots after factorization): 1 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 43 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 156 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 43 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 156 INFOG(20) (estimated number of entries in the factors): 306174 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 42 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 150 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 linear system matrix = precond matrix: Matrix Object: 4 MPI processes type: mpiaij rows=894, cols=894 total: nonzeros=263360, allocated nonzeros=263360 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 47 nodes, limit used is 5 >> # of iterations: 2 KSPConvergedReason: 3 Entering ZMUMPS driver with JOB, N, NZ = -2 894 263360 Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley Sent: Tuesday, October 23, 2012 4:30 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 7:17 PM, Jinquan Zhong > wrote: Thanks, Jed. Any way to get the condition number. I used -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 It didn?t work. This is not helpful. Would you understand this if someone mailed you "It didn't work"? Send the output. Matt Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown Sent: Tuesday, October 23, 2012 3:28 PM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote: That is new for me. What would you suggest, Matt? Were you using LU or Cholesky before? That is the difference between -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps to choose MUMPS. You can access the MUMPS options with -mat_mumps_icntl_opaquenumber. It looks like PETSc's Clique interface does not work in parallel. (A student was working on it recently and it seems to work in serial.) When it is fixed to work in parallel, that is almost certainly what you should use. Alternatively, there may well be a Fast method, depending on the structure of the system and that fat dense block. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 23 19:02:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 23 Oct 2012 20:02:47 -0400 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <7237384C7A8F2642A8854B751619FA31704EE41F@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3D0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3DF@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3FD@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE40D@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE41F@exchange.scsolutions.com> Message-ID: On Tue, Oct 23, 2012 at 7:42 PM, Jinquan Zhong wrote: > I am seeing **** > > ** ** > > 652 KSP Residual norm 6.568962571627e+02 % max 1.464924876151e+14 min > 8.431983181754e+03 max/min 1.737343214015e+10**** > > 653 KSP Residual norm 6.562904211497e+02 % max 1.464924876151e+14 min > 8.431776971754e+03 max/min 1.737385702988e+10**** > > ** ** > > when I used**** > > -pc_type none**** > > ** ** > > Does that indicate the Cond# is 1.7x10e10? > Yes, and that is probably an underestimate. Matt > > > Jinquan**** > > ** ** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Tuesday, October 23, 2012 4:38 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > ** ** > > On Tue, Oct 23, 2012 at 6:34 PM, Jinquan Zhong > wrote:**** > > * > That is true. I used***** > > **** > > -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view > -mat_mumps_icntl_4 2 -mat_mumps_icntl_5 0 -mat_mumps_icntl_18 3 > -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 -ksp_type gmres > -ksp_monitor_singular_value -ksp_gmres_restart 1000**** > > ** ** > > Read my reply about using GMRES to estimate condition number _while_ using > a preconditioner.**** > > **** > > **** > > **** > > *Here is the info I got from the setting I had. I didn?t see the > condition number appeared:***** > > **** > > **** > > Entering ZMUMPS driver with JOB, N, NZ = 1 894 0*** > * > > **** > > ZMUMPS 4.10.0 **** > > L U Solver for unsymmetric matrices**** > > Type of parallelism: Working host**** > > **** > > ****** ANALYSIS STEP ************ > > **** > > ** Max-trans not allowed because matrix is distributed**** > > Using ParMETIS for parallel ordering.**** > > Structual symmetry is:100%**** > > WARNING: Largest root node of size 173 not selected for > parallel execution**** > > **** > > Leaving analysis phase with ...**** > > INFOG(1) = 0**** > > INFOG(2) = 0**** > > -- (20) Number of entries in factors (estim.) = 306174**** > > -- (3) Storage of factors (REAL, estimated) = 306174**** > > -- (4) Storage of factors (INT , estimated) = 7102**** > > -- (5) Maximum frontal size (estimated) = 495**** > > -- (6) Number of nodes in the tree = 66**** > > -- (32) Type of analysis effectively used = 2**** > > -- (7) Ordering option effectively used = 2**** > > ICNTL(6) Maximum transversal option = 0**** > > ICNTL(7) Pivot order option = 7**** > > Percentage of memory relaxation (effective) = 25**** > > Number of level 2 nodes = 0**** > > Number of split nodes = 0**** > > RINFOG(1) Operations during elimination (estim)= 8.929D+07**** > > Distributed matrix entry format (ICNTL(18)) = 3**** > > ** Rank of proc needing largest memory in IC facto : 1**** > > ** Estimated corresponding MBYTES for IC facto : 43**** > > ** Estimated avg. MBYTES per work. proc at facto (IC) : 39**** > > ** TOTAL space in MBYTES for IC factorization : 156**** > > ** Rank of proc needing largest memory for OOC facto : 1**** > > ** Estimated corresponding MBYTES for OOC facto : 46**** > > ** Estimated avg. MBYTES per work. proc at facto (OOC) : 41**** > > ** TOTAL space in MBYTES for OOC factorization : 165**** > > Entering ZMUMPS driver with JOB, N, NZ = 2 894 263360*** > * > > **** > > ****** FACTORIZATION STEP ************ > > **** > > **** > > GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...**** > > NUMBER OF WORKING PROCESSES = 4**** > > OUT-OF-CORE OPTION (ICNTL(22)) = 0**** > > REAL SPACE FOR FACTORS = 306174**** > > INTEGER SPACE FOR FACTORS = 7102**** > > MAXIMUM FRONTAL SIZE (ESTIMATED) = 495**** > > NUMBER OF NODES IN THE TREE = 66**** > > Convergence error after scaling for ONE-NORM (option 7/8) = 0.31D+00**** > > Maximum effective relaxed size of S = 306300**** > > Average effective relaxed size of S = 185353**** > > **** > > REDISTRIB: TOTAL DATA LOCAL/SENT = 68276 195084**** > > GLOBAL TIME FOR MATRIX DISTRIBUTION = 0.0145**** > > ** Memory relaxation parameter ( ICNTL(14) ) : 25**** > > ** Rank of processor needing largest memory in facto : 1**** > > ** Space in MBYTES used by this processor for facto : 43**** > > ** Avg. Space in MBYTES per working proc during facto : 39**** > > **** > > ELAPSED TIME FOR FACTORIZATION = 0.0862**** > > Maximum effective space used in S (KEEP8(67) = 245025**** > > Average effective space used in S (KEEP8(67) = 119077**** > > ** EFF Min: Rank of processor needing largest memory : 1**** > > ** EFF Min: Space in MBYTES used by this processor : 42**** > > ** EFF Min: Avg. Space in MBYTES per working proc : 37**** > > **** > > GLOBAL STATISTICS **** > > RINFOG(2) OPERATIONS IN NODE ASSEMBLY = 2.372D+05**** > > ------(3) OPERATIONS IN NODE ELIMINATION= 8.929D+07**** > > INFOG (9) REAL SPACE FOR FACTORS = 306184**** > > INFOG(10) INTEGER SPACE FOR FACTORS = 7104**** > > INFOG(11) MAXIMUM FRONT SIZE = 495**** > > INFOG(29) NUMBER OF ENTRIES IN FACTORS = 306184**** > > INFOG(12) NB OF OFF DIAGONAL PIVOTS = 29**** > > INFOG(13) NUMBER OF DELAYED PIVOTS = 1**** > > INFOG(14) NUMBER OF MEMORY COMPRESS = 0**** > > KEEP8(108) Extra copies IP stacking = 0**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > **** > > **** > > ****** SOLVE & CHECK STEP ************ > > **** > > **** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 0 KSP Residual norm 1.890433086271e+01 % max 1.000000000000e+00 min > 1.000000000000e+00 max/min 1.000000000000e+00**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > **** > > **** > > ****** SOLVE & CHECK STEP ************ > > **** > > **** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 1 KSP Residual norm 1.804170434909e-06 % max 1.000000180789e+00 min > 1.000000180789e+00 max/min 1.000000000000e+00**** > > Entering ZMUMPS driver with JOB, N, NZ = 3 894 263360*** > * > > **** > > **** > > ****** SOLVE & CHECK STEP ************ > > **** > > **** > > STATISTICS PRIOR SOLVE PHASE ...........**** > > NUMBER OF RIGHT-HAND-SIDES = 1**** > > BLOCKING FACTOR FOR MULTIPLE RHS = 1**** > > ICNTL (9) = 1**** > > --- (10) = 0**** > > --- (11) = 0**** > > --- (20) = 0**** > > --- (21) = 1**** > > --- (30) = 0**** > > ** Rank of processor needing largest memory in solve : 1**** > > ** Space in MBYTES used by this processor for solve : 6**** > > ** Avg. Space in MBYTES per working proc during solve : 3**** > > 2 KSP Residual norm 4.466758995607e-13 % max 1.000000467998e+00 min > 9.999992798573e-01 max/min 1.000001188141e+00**** > > ** ** > > Here are your estimates of max and min singular values, and their ratio > (the condition number). These are for the *preconditioned operator*. > Since you use a direct solver, it is almost exactly 1.**** > > **** > > KSP Object: 4 MPI processes**** > > type: gmres**** > > GMRES: restart=1000, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement**** > > GMRES: happy breakdown tolerance 1e-30**** > > maximum iterations=10000, initial guess is zero**** > > tolerances: relative=1e-12, absolute=1e-12, divergence=10000**** > > left preconditioning**** > > using PRECONDITIONED norm type for convergence test**** > > PC Object: 4 MPI processes**** > > type: lu**** > > LU: out-of-place factorization**** > > tolerance for zero pivot 2.22045e-14**** > > matrix ordering: natural**** > > factor fill ratio given 0, needed 0**** > > Factored matrix follows:**** > > Matrix Object: 4 MPI processes**** > > type: mpiaij**** > > rows=894, cols=894**** > > package used to perform factorization: mumps**** > > total: nonzeros=306174, allocated nonzeros=306174**** > > total number of mallocs used during MatSetValues calls =0**** > > MUMPS run parameters:**** > > SYM (matrix type): 0 **** > > PAR (host participation): 1 **** > > ICNTL(1) (output for error): 6 **** > > ICNTL(2) (output of diagnostic msg): 0 **** > > ICNTL(3) (output for global info): 6 **** > > ICNTL(4) (level of printing): 2 **** > > ICNTL(5) (input mat struct): 0 **** > > ICNTL(6) (matrix prescaling): 7 **** > > ICNTL(7) (sequentia matrix ordering):7 **** > > ICNTL(8) (scalling strategy): 77 **** > > ICNTL(10) (max num of refinements): 0 **** > > ICNTL(11) (error analysis): 0 **** > > ICNTL(12) (efficiency control): 1 ** > ** > > ICNTL(13) (efficiency control): 0 ** > ** > > ICNTL(14) (percentage of estimated workspace increase): 20 * > *** > > ICNTL(18) (input mat struct): 3 ** > ** > > ICNTL(19) (Shur complement info): 0 ** > ** > > ICNTL(20) (rhs sparse pattern): 0 ** > ** > > ICNTL(21) (solution struct): 1 ** > ** > > ICNTL(22) (in-core/out-of-core facility): 0 ** > ** > > ICNTL(23) (max size of memory can be allocated locally):0 ** > ** > > ICNTL(24) (detection of null pivot rows): 0 ** > ** > > ICNTL(25) (computation of a null space basis): 0 ** > ** > > ICNTL(26) (Schur options for rhs or solution): 0 ** > ** > > ICNTL(27) (experimental parameter): -8 * > *** > > ICNTL(28) (use parallel or sequential ordering): 2 ** > ** > > ICNTL(29) (parallel ordering): 2 ** > ** > > ICNTL(30) (user-specified set of entries in inv(A)): 0 ** > ** > > ICNTL(31) (factors is discarded in the solve phase): 0 ** > ** > > ICNTL(33) (compute determinant): 0 ** > ** > > CNTL(1) (relative pivoting threshold): 0.01 **** > > CNTL(2) (stopping criterion of refinement): 1.49012e-08 **** > > CNTL(3) (absolute pivoting threshold): 0 **** > > CNTL(4) (value of static pivoting): -1 **** > > CNTL(5) (fixation for null pivots): 0 **** > > RINFO(1) (local estimated flops for the elimination after > analysis): **** > > [0] 4.25638e+06 **** > > [1] 7.22129e+07 **** > > [2] 8.01308e+06 **** > > [3] 4.81122e+06 **** > > RINFO(2) (local estimated flops for the assembly after > factorization): **** > > [0] 68060 **** > > [1] 5860 **** > > [2] 83570 **** > > [3] 79676 **** > > RINFO(3) (local estimated flops for the elimination after > factorization): **** > > [0] 4.25638e+06 **** > > [1] 7.2213e+07 **** > > [2] 8.01308e+06 **** > > [3] 4.81122e+06 **** > > INFO(15) (estimated size of (in MB) MUMPS internal data for > running numerical factorization): **** > > [0] 37 **** > > [1] 43 **** > > [2] 38 **** > > [3] 38 **** > > INFO(16) (size of (in MB) MUMPS internal data used during > numerical factorization): **** > > [0] 37 **** > > [1] 43 **** > > [2] 38 **** > > [3] 38 **** > > INFO(23) (num of pivots eliminated on this processor after > factorization): **** > > [0] 350 **** > > [1] 319 **** > > [2] 134 **** > > [3] 91 **** > > RINFOG(1) (global estimated flops for the elimination after > analysis): 8.92936e+07 **** > > RINFOG(2) (global estimated flops for the assembly after > factorization): 237166 **** > > RINFOG(3) (global estimated flops for the elimination after > factorization): 8.92937e+07 **** > > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): > (0,0)*(2^0)**** > > INFOG(3) (estimated real workspace for factors on all > processors after analysis): 306174 **** > > INFOG(4) (estimated integer workspace for factors on all > processors after analysis): 7102 **** > > INFOG(5) (estimated maximum front size in the complete > tree): 495 **** > > INFOG(6) (number of nodes in the complete tree): 66 **** > > INFOG(7) (ordering option effectively use after analysis): 2 > **** > > INFOG(8) (structural symmetry in percent of the permuted > matrix after analysis): 100 **** > > INFOG(9) (total real/complex workspace to store the matrix > factors after factorization): 306184 **** > > INFOG(10) (total integer space store the matrix factors > after factorization): 7104 **** > > INFOG(11) (order of largest frontal matrix after > factorization): 495 **** > > INFOG(12) (number of off-diagonal pivots): 29 **** > > INFOG(13) (number of delayed pivots after factorization): 1 > **** > > INFOG(14) (number of memory compress after factorization): 0 > **** > > INFOG(15) (number of steps of iterative refinement after > solution): 0 **** > > INFOG(16) (estimated size (in MB) of all MUMPS internal data > for factorization after analysis: value on the most memory consuming > processor): 43 **** > > INFOG(17) (estimated size of all MUMPS internal data for > factorization after analysis: sum over all processors): 156 **** > > INFOG(18) (size of all MUMPS internal data allocated during > factorization: value on the most memory consuming processor): 43 **** > > INFOG(19) (size of all MUMPS internal data allocated during > factorization: sum over all processors): 156 **** > > INFOG(20) (estimated number of entries in the factors): > 306174 **** > > INFOG(21) (size in MB of memory effectively used during > factorization - value on the most memory consuming processor): 42 **** > > INFOG(22) (size in MB of memory effectively used during > factorization - sum over all processors): 150 **** > > INFOG(23) (after analysis: value of ICNTL(6) effectively > used): 0 **** > > INFOG(24) (after analysis: value of ICNTL(12) effectively > used): 1 **** > > INFOG(25) (after factorization: number of pivots modified by > static pivoting): 0 **** > > linear system matrix = precond matrix:**** > > Matrix Object: 4 MPI processes**** > > type: mpiaij**** > > rows=894, cols=894**** > > total: nonzeros=263360, allocated nonzeros=263360**** > > total number of mallocs used during MatSetValues calls =0**** > > using I-node (on process 0) routines: found 47 nodes, limit used is 5 > **** > > >> # of iterations: 2**** > > KSPConvergedReason: 3**** > > Entering ZMUMPS driver with JOB, N, NZ = -2 894 263360*** > * > > **** > > **** > > Jinquan**** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley > *Sent:* Tuesday, October 23, 2012 4:30 PM**** > > > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > **** > > On Tue, Oct 23, 2012 at 7:17 PM, Jinquan Zhong > wrote:**** > > Thanks, Jed.**** > > **** > > Any way to get the condition number. I used **** > > **** > > -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000**** > > **** > > It didn?t work.**** > > **** > > This is not helpful. Would you understand this if someone mailed you "It > didn't work"? Send the output.**** > > **** > > Matt**** > > **** > > **** > > Jinquan**** > > **** > > *From:* petsc-users-bounces at mcs.anl.gov [mailto: > petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Jed Brown > *Sent:* Tuesday, October 23, 2012 3:28 PM > *To:* PETSc users list > *Subject:* Re: [petsc-users] Setting up MUMPS in PETSc**** > > **** > > On Tue, Oct 23, 2012 at 5:21 PM, Jinquan Zhong > wrote:**** > > That is new for me. What would you suggest, Matt?**** > > **** > > Were you using LU or Cholesky before? That is the difference between > -pc_type lu and -pc_type cholesky. Use -pc_factor_mat_solver_package mumps > to choose MUMPS. You can access the MUMPS options with > -mat_mumps_icntl_opaquenumber.**** > > **** > > It looks like PETSc's Clique interface does not work in parallel. (A > student was working on it recently and it seems to work in serial.) When it > is fixed to work in parallel, that is almost certainly what you should use. > Alternatively, there may well be a Fast method, depending on the structure > of the system and that fat dense block.**** > > > > **** > > **** > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener**** > > ** ** > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Oct 23 22:49:26 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 23 Oct 2012 22:49:26 -0500 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> Message-ID: Clique-petsc interface does not work yet. Things need to be done: - reuse symbolic factor (Jack added this feature several weeks ago, but I do not have time work on it yet) - distributed rhs and solution vector (not sure about clique's status) Jinquan: did you use Cholesky or LU from mumps? For symmetric problem, use mumps Cholesky. Hong On Tue, Oct 23, 2012 at 4:14 PM, Jed Brown wrote: > On Tue, Oct 23, 2012 at 2:48 PM, Jinquan Zhong wrote: > >> >> Soil-structure interaction. >> > > Why is it dense? Is it effectively the solution of another equation? An > integral operator? > > >> >> >> ** >> >> The amount of fill depends on the size of minimal vertex separators. >> Sparse matrices with the same number of nonzeros and same number of >> nonzeros per row can have vertex separators that are orders of magnitude >> different in size. The fill is quadratic in the size of the separators and >> computation is cubic.**** >> >> >> Could you be more specific? I am not quite with you yet. >> > > A 10x10x10000 3D problem with hex elements has about 1M dofs and about 27 > nonzeros per row. The minimal vertex separator consists of 10^2 vertices, > so the final dense matrix is 100x100. The direct solver is extremely fast > for this problem. > > A 100x100x100 3D problem has the same number of dofs and nonzeros per row. > The minimal vertex separator is 100^2 vertices, so the final dense matrix > is 10000x10000 (and there are many pretty big dense matrices to get there). > This problem requires on the order of 10000 times as much memory and > 1000000 times as many flops. > > If we switch from a FEM discretization to a FD discretization with a > stencil width of 3, the vertex separator grows by a factor of 3, increasing > memory usage by a factor of 9 and flops by a factor of 27. If you replace > that high order system with high order continuous Galerkin FEM using larger > elements so the number of dofs is constant, the number of nonzeros in the > matrix may grow, but the vertex separators go back to being the same as the > original problem. > > >> **** >> >> ** ** >> >> Is your problem symmetric?**** >> >> ** ** >> >> >> Yes. >> > > Hong, is the Clique code in petsc-dev ready to use? > > There is no reason we should keep spending time dealing with MUMPS quirks > and scalability problems on symmetric problems. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Wed Oct 24 04:10:11 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Wed, 24 Oct 2012 11:10:11 +0200 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> Message-ID: <5087B073.40003@gfz-potsdam.de> Jinquan, As a rule of thumb, you should always try to run MUMPS on as few nodes as possible which have as much memory as possible. Given the same matrix, MUMPS requires more memory when run with, for instance, 4 process than with 2. You gain nothing trying thousands of processes without even looking at statistics mumps gives you (through its output). If your matrix is symmetric you should use cholesky (LDL^T) factorization in MUMPS. This will save a lot of memory and time. To estimate conditioning of the problem you may also consider the algorithm described here: http://scicomp.stackexchange.com/questions/3486/what-are-the-symptoms-of-ill-conditioning-when-using-direct-methods It is easy to implement it using MUMPS. -- Regards, Alexander From thomas.witkowski at tu-dresden.de Wed Oct 24 14:23:48 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 24 Oct 2012 21:23:48 +0200 Subject: [petsc-users] How to access the KSP objects from a PCFIELDSPLIT Schur preconditioner Message-ID: <50884044.6030405@tu-dresden.de> I want to play around with some special type of a Schur complement preconditioner. It perfectly fits into PETSc's fieldsplit preconditioner (schur - full factorization). But for the inner solvers, I would like to create some MatShells. So I need to access both KSP objects, for the upper block (KSP(A_00)) and for the Schur complement (KSP(S)). How can I get these objects in my source code? Best regards, Thomas From knepley at gmail.com Wed Oct 24 14:33:47 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Oct 2012 15:33:47 -0400 Subject: [petsc-users] How to access the KSP objects from a PCFIELDSPLIT Schur preconditioner In-Reply-To: <50884044.6030405@tu-dresden.de> References: <50884044.6030405@tu-dresden.de> Message-ID: On Wed, Oct 24, 2012 at 3:23 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > I want to play around with some special type of a Schur complement > preconditioner. It perfectly fits into PETSc's fieldsplit preconditioner > (schur - full factorization). But for the inner solvers, I would like to > create some MatShells. So I need to access both KSP objects, for the upper > block (KSP(A_00)) and for the Schur complement (KSP(S)). How can I get > these objects in my source code? > First you have to make them different. This happens when you give options for -fieldsplit_1_upper_* -fieldsplit_1_inner_* This is how I customize the solvers to get things like SIMPLE. Once they are created, you can get them by pulling apart the FS. The upper is fs->kspupper, and the inner is the KSP from the MATSCHUR from fs->schur. Matt > Best regards, > > Thomas > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Wed Oct 24 15:04:04 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Wed, 24 Oct 2012 22:04:04 +0200 Subject: [petsc-users] How to access the KSP objects from a PCFIELDSPLIT Schur preconditioner In-Reply-To: References: <50884044.6030405@tu-dresden.de> Message-ID: <508849B4.8010605@tu-dresden.de> Am 24.10.2012 21:33, schrieb Matthew Knepley: > On Wed, Oct 24, 2012 at 3:23 PM, Thomas Witkowski > > wrote: > > I want to play around with some special type of a Schur complement > preconditioner. It perfectly fits into PETSc's fieldsplit > preconditioner (schur - full factorization). But for the inner > solvers, I would like to create some MatShells. So I need to > access both KSP objects, for the upper block (KSP(A_00)) and for > the Schur complement (KSP(S)). How can I get these objects in my > source code? > > > First you have to make them different. This happens when you give > options for > > -fieldsplit_1_upper_* > -fieldsplit_1_inner_* > > This is how I customize the solvers to get things like SIMPLE. Once > they are created, you > can get them by pulling apart the FS. The upper is fs->kspupper, and > the inner is the KSP > from the MATSCHUR from fs->schur. I don't understand your solution. Maybe I should put some lines of code to emphasize my question. The relevant part looks as follows: PCSetType(pc, PCFIELDSPLIT); PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_FULL); ... PCFieldSplitSetIS(pc, "velocity", is0); PCFieldSplitSetIS(pc, "pressure", is1); ... KSP velocity; KSP schur; .... Okay, I don't solve any Stokes/Navier Stokes equation, but to make the names somehow convenient I choose here the names velocity and pressure for the two system blocks. My question is now, how to figure out the two inner solver objects from the field split preconditioner. Matt, if your answer is still appropriate to my question, can you make it clear on the above piece of code? Thank you very much. Thomas > > Matt > > Best regards, > > Thomas > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Wed Oct 24 15:19:06 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Wed, 24 Oct 2012 15:19:06 -0500 Subject: [petsc-users] How to access the KSP objects from a PCFIELDSPLIT Schur preconditioner In-Reply-To: <508849B4.8010605@tu-dresden.de> References: <50884044.6030405@tu-dresden.de> <508849B4.8010605@tu-dresden.de> Message-ID: You are looking for PCFieldSplitGetSubKSP(). Matt's comment was about distinguishing between the KSP living inside of S = D - C A^{-1} B and the one used directly by PCFIeldSplit when solving with A. On Wed, Oct 24, 2012 at 3:04 PM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Am 24.10.2012 21:33, schrieb Matthew Knepley: > > On Wed, Oct 24, 2012 at 3:23 PM, Thomas Witkowski < > thomas.witkowski at tu-dresden.de> wrote: > >> I want to play around with some special type of a Schur complement >> preconditioner. It perfectly fits into PETSc's fieldsplit preconditioner >> (schur - full factorization). But for the inner solvers, I would like to >> create some MatShells. So I need to access both KSP objects, for the upper >> block (KSP(A_00)) and for the Schur complement (KSP(S)). How can I get >> these objects in my source code? >> > > First you have to make them different. This happens when you give > options for > > -fieldsplit_1_upper_* > -fieldsplit_1_inner_* > > This is how I customize the solvers to get things like SIMPLE. Once they > are created, you > can get them by pulling apart the FS. The upper is fs->kspupper, and the > inner is the KSP > from the MATSCHUR from fs->schur. > > I don't understand your solution. Maybe I should put some lines of code to > emphasize my question. The relevant part looks as follows: > > PCSetType(pc, PCFIELDSPLIT); > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_FULL); > ... > PCFieldSplitSetIS(pc, "velocity", is0); > PCFieldSplitSetIS(pc, "pressure", is1); > ... > KSP velocity; > KSP schur; > .... > > Okay, I don't solve any Stokes/Navier Stokes equation, but to make the > names somehow convenient I choose here the names velocity and pressure for > the two system blocks. My question is now, how to figure out the two inner > solver objects from the field split preconditioner. > > Matt, if your answer is still appropriate to my question, can you make it > clear on the above piece of code? Thank you very much. > > Thomas > > > > Matt > > >> Best regards, >> >> Thomas >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Wed Oct 24 18:11:08 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Wed, 24 Oct 2012 23:11:08 +0000 Subject: [petsc-users] Setting up MUMPS in PETSc In-Reply-To: <5087B073.40003@gfz-potsdam.de> References: <7237384C7A8F2642A8854B751619FA31704EE34C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3A0@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704EE3B7@exchange.scsolutions.com> , <5087B073.40003@gfz-potsdam.de> Message-ID: <7237384C7A8F2642A8854B751619FA31704EF8A0@exchange.scsolutions.com> Thanks, Alexander. That explains the phenomena I needed to use 1152 cores for 576 partitions and only 432 cores for 144 partitions. I will come up with more questions for you. Jinquan ________________________________________ From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Alexander Grayver [agrayver at gfz-potsdam.de] Sent: Wednesday, October 24, 2012 2:10 AM To: PETSc users list Subject: Re: [petsc-users] Setting up MUMPS in PETSc Jinquan, As a rule of thumb, you should always try to run MUMPS on as few nodes as possible which have as much memory as possible. Given the same matrix, MUMPS requires more memory when run with, for instance, 4 process than with 2. You gain nothing trying thousands of processes without even looking at statistics mumps gives you (through its output). If your matrix is symmetric you should use cholesky (LDL^T) factorization in MUMPS. This will save a lot of memory and time. To estimate conditioning of the problem you may also consider the algorithm described here: http://scicomp.stackexchange.com/questions/3486/what-are-the-symptoms-of-ill-conditioning-when-using-direct-methods It is easy to implement it using MUMPS. -- Regards, Alexander From gokhalen at gmail.com Wed Oct 24 18:32:00 2012 From: gokhalen at gmail.com (Nachiket Gokhale) Date: Wed, 24 Oct 2012 19:32:00 -0400 Subject: [petsc-users] MatCreateComposite Message-ID: I have matrices K and M and I am trying to create the matrix D = K - (w^2) M where w is a scalar. I am trying to do this using matscale of M and then using MatComposite and MatCompositeMerge. However, I am not clear about the documentation of MatCreateComposite. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateComposite.html It says the output parameter is A, however I don't see A in the call. Perhaps the output is Mat? Also, the documentation says that nmat is the number of matrices to put in, but I see only one argument - does this function take variable number of arguments? I haven't seen Petsc functions like this in the past ... Also, the call in the example seems ex9.c is 67: MatCreateComposite(PETSC_COMM_WORLD,3,A,&B); So, there are three matrices to put in but only one (A) is put in. I don't quite understand this either. Any help would be appreciated, Thanks, -Nachiket From knepley at gmail.com Wed Oct 24 18:38:19 2012 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Oct 2012 19:38:19 -0400 Subject: [petsc-users] MatCreateComposite In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 7:32 PM, Nachiket Gokhale wrote: > I have matrices K and M and I am trying to create the matrix > > D = K - (w^2) M where w is a scalar. > > I am trying to do this using matscale of M and then using MatComposite > and MatCompositeMerge. However, I am not clear about the > documentation of MatCreateComposite. > > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateComposite.html > > It says the output parameter is A, however I don't see A in the call. > Perhaps the output is Mat? Also, the documentation says that nmat is > the number of matrices to put in, but I see only one argument - does > this function take variable number of arguments? I haven't seen Petsc > functions like this in the past ... > The argument mats in an array. That is why you need nmat. You put both K and M in the array. Matt > > Also, the call in the example seems ex9.c is > > 67: MatCreateComposite(PETSC_COMM_WORLD,3,A,&B); > > So, there are three matrices to put in but only one (A) is put in. I > don't quite understand this either. > > Any help would be appreciated, > > Thanks, > > -Nachiket > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Thu Oct 25 05:39:03 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 25 Oct 2012 12:39:03 +0200 Subject: [petsc-users] How to access the KSP objects from a PCFIELDSPLIT Schur preconditioner In-Reply-To: References: <50884044.6030405@tu-dresden.de> <508849B4.8010605@tu-dresden.de> Message-ID: <508916C7.7050809@tu-dresden.de> Thanks, exactly what I've searched for. Thomas Am 24.10.2012 22:19, schrieb Jed Brown: > You are looking for PCFieldSplitGetSubKSP(). > > Matt's comment was about distinguishing between the KSP living inside > of S = D - C A^{-1} B and the one used directly by PCFIeldSplit when > solving with A. > > On Wed, Oct 24, 2012 at 3:04 PM, Thomas Witkowski > > wrote: > > Am 24.10.2012 21:33, schrieb Matthew Knepley: >> On Wed, Oct 24, 2012 at 3:23 PM, Thomas Witkowski >> > > wrote: >> >> I want to play around with some special type of a Schur >> complement preconditioner. It perfectly fits into PETSc's >> fieldsplit preconditioner (schur - full factorization). But >> for the inner solvers, I would like to create some MatShells. >> So I need to access both KSP objects, for the upper block >> (KSP(A_00)) and for the Schur complement (KSP(S)). How can I >> get these objects in my source code? >> >> >> First you have to make them different. This happens when you give >> options for >> >> -fieldsplit_1_upper_* >> -fieldsplit_1_inner_* >> >> This is how I customize the solvers to get things like SIMPLE. >> Once they are created, you >> can get them by pulling apart the FS. The upper is fs->kspupper, >> and the inner is the KSP >> from the MATSCHUR from fs->schur. > I don't understand your solution. Maybe I should put some lines of > code to emphasize my question. The relevant part looks as follows: > > PCSetType(pc, PCFIELDSPLIT); > PCFieldSplitSetSchurFactType(pc, PC_FIELDSPLIT_SCHUR_FACT_FULL); > ... > PCFieldSplitSetIS(pc, "velocity", is0); > PCFieldSplitSetIS(pc, "pressure", is1); > ... > KSP velocity; > KSP schur; > .... > > Okay, I don't solve any Stokes/Navier Stokes equation, but to make > the names somehow convenient I choose here the names velocity and > pressure for the two system blocks. My question is now, how to > figure out the two inner solver objects from the field split > preconditioner. > > Matt, if your answer is still appropriate to my question, can you > make it clear on the above piece of code? Thank you very much. > > Thomas > > >> >> Matt >> >> Best regards, >> >> Thomas >> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.witkowski at tu-dresden.de Thu Oct 25 06:49:16 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Thu, 25 Oct 2012 13:49:16 +0200 Subject: [petsc-users] KSPSolve with both vectors being the same and MatShell Message-ID: <5089273C.1030600@tu-dresden.de> Following the documentation it is allowed to set the rhs and solution vectors in KSPSolve to be the same. When my KSPSolve is based on a MatShell, must the MatShell operator function be implemented to handle also the case that both vectors are the same? Or can I assume, within MatShell, that both vectors, which are provided by KSPSolve, are really two different objects? Thomas From jedbrown at mcs.anl.gov Thu Oct 25 07:29:57 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 25 Oct 2012 07:29:57 -0500 Subject: [petsc-users] KSPSolve with both vectors being the same and MatShell In-Reply-To: <5089273C.1030600@tu-dresden.de> References: <5089273C.1030600@tu-dresden.de> Message-ID: On Thu, Oct 25, 2012 at 6:49 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Following the documentation it is allowed to set the rhs and solution > vectors in KSPSolve to be the same. When my KSPSolve is based on a > MatShell, must the MatShell operator function be implemented to handle also > the case that both vectors are the same? Or can I assume, within MatShell, > that both vectors, which are provided by KSPSolve, are really two different > objects? This is taken care of by KSPSolve, so the vectors are always different in the implementations. Note that MatMult is typically used with different vectors internal to the KSP. The MatMult documentation requires the vectors to be different. > > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gregor.Matura at dlr.de Thu Oct 25 08:43:24 2012 From: Gregor.Matura at dlr.de (Gregor.Matura at dlr.de) Date: Thu, 25 Oct 2012 13:43:24 +0000 Subject: [petsc-users] Block matrix data layout and access Message-ID: <2ED7073FE68F3F40B2D8739A46A7971B01537D28@dlrexmbx02.intra.dlr.de> Hi! I'm rather new to PETSc and got along on my own so far. However, I've now run into the situation that the tutorial does not provide enough detail and on the other side going through every example found by grep with the possibly right command would take almost forever. So I would be very happy to get some hints where to start. My problem: read in a matrix in a special format, partition and redistribute it, solve. So far, I've read in my matrix, transformed it according to the input of MatCreateMPIAIJWithSplitArrays() and finally, for a start, KSPSolve. Here comes the 'but': my matrix does have a "block" format: |a ... a| |a ... a| |b ... b| |b ... b| |c ... c| |c ... c| The matrix is sparse (by far not every x is set), every a, b, c, ... is a small 5x5 block, each 25 double data values of a block are saved consecutively, processor 1 holds every a, #2 every b and so on. PETSc tutorial says (as far as I understood) that PETSc's blocks are _logically_ different, but are stored just this way. And: in some sense DMDA could be the right way to access matrix data layout as PETSc uses it (would be good to achieve best possible performance). This said, my precise questions are: Is DMDACreate2D the right way to go? Does it match my data layout best with PETSc's layout? Which example file is suited best to read up on? Does this really speeds up time to solution or should I stick with the transformed non-block variant? TIA, Gregor Matura -------------------------- Deutsches Zentrum f?r Luft- und Raumfahrt e.V. (DLR) German Aerospace Center -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Thu Oct 25 08:52:23 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Thu, 25 Oct 2012 08:52:23 -0500 Subject: [petsc-users] Block matrix data layout and access In-Reply-To: <2ED7073FE68F3F40B2D8739A46A7971B01537D28@dlrexmbx02.intra.dlr.de> References: <2ED7073FE68F3F40B2D8739A46A7971B01537D28@dlrexmbx02.intra.dlr.de> Message-ID: On Thu, Oct 25, 2012 at 8:43 AM, wrote: > Hi!**** > > ** ** > > I'm rather new to PETSc and got along on my own so far. However, I've now > run into the situation that the tutorial does not provide enough detail and > on the other side going through every example found by grep with the > possibly right command would take almost forever. So I would be very happy > to get some hints where to start.**** > > ** ** > > My problem: read in a matrix in a special format, partition and > redistribute it, solve.**** > > ** ** > > So far, I've read in my matrix, transformed it according to the input of > MatCreateMPIAIJWithSplitArrays() and finally, for a start, KSPSolve. > Strongly recommend using MatSetValues[Blocked][Local]() instead of MatCreateMPIAIJWithSplitArrays(). Just preallocate and set the values from wherever you know them. This way, the same code will work immediately with different matrix formats and it's easier to transition to generating the matrix on-the-fly (rather than reading from disk which is always a bottleneck). > **** > > ** ** > > Here comes the 'but': my matrix does have a "block" format:**** > > ** ** > > |a ... a|**** > > |a ... a|**** > > |b ... b|**** > > |b ... b|**** > > |c ... c|**** > > |c ... c|**** > > ** ** > > The matrix is sparse (by far not every x is set), every a, b, c, ... is a > small 5x5 block, each 25 double data values of a block are saved > consecutively, processor 1 holds every a, #2 every b and so on.**** > > ** ** > > PETSc tutorial says (as far as I understood) that PETSc's blocks are > _logically_ different, but are stored just this way. > Not sure which routine you are looking at. MATMPI*B*AIJ is a good format for systems like this. When the method you intend to use is supported by BAIJ matrices, they are generally faster by 30% or more. By using MatSetValues-style matrix assembly, you can switch format a run-time. > And: in some sense DMDA could be the right way to access matrix data > layout as PETSc uses it (would be good to achieve best possible > performance). > DMDA is a structured grid abstraction. Don't confuse 2D structured grids with matrices. > **** > > ** ** > > This said, my precise questions are: Is DMDACreate2D the right way to go? > Does it match my data layout best with PETSc's layout? Which example file > is suited best to read up on? Does this really speeds up time to solution > or should I stick with the transformed non-block variant?**** > > ** ** > > TIA,**** > > ** ** > > Gregor Matura**** > > ** ** > > ??????????????????????????**** > > *Deutsches Zentrum f?r Luft- und Raumfahrt* e.V. (DLR)**** > > German Aerospace Center**** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l.buikstra at student.utwente.nl Thu Oct 25 09:12:31 2012 From: l.buikstra at student.utwente.nl (l.buikstra at student.utwente.nl) Date: Thu, 25 Oct 2012 14:12:31 +0000 Subject: [petsc-users] Output format of FFTW Message-ID: Dear all, I'm trying to get a feel for fourier transforms in Petsc, but I can find very little information about doing FTs through Petsc's interface. The examples all seem to just do a transform, then transform back to confirm the same function comes out again; that works fine over here, but I can't seem to figure out how the actual fourier transform itself is stored so it can be manipulated or used before transforming back. I wrote a short test program: http://pastebin.com/4saWgqLu It fills a vector with a sine, then transforms it and displays the result, which looks like this: http://imageshack.us/a/img37/1135/whatph.png (the original vector is 2000 points; I expect this is 4000 because it somehow represents a complex vector?) I would expect a smooth function with two peaks on the far left and right, but instead all it shows is a repetitively garbled mess in the first half of the output and mostly zeroes in the second. Am I just doing something wrong? If so, what, and if not, what exactly does this output represent? How does it relate to the output I'm expecting? Any help would be greatly appreciated. Yours, Leon Buikstra From knepley at gmail.com Thu Oct 25 09:18:37 2012 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Oct 2012 10:18:37 -0400 Subject: [petsc-users] Output format of FFTW In-Reply-To: References: Message-ID: On Thu, Oct 25, 2012 at 10:12 AM, wrote: > Dear all, > > I'm trying to get a feel for fourier transforms in Petsc, but I can find > very little information about doing FTs through Petsc's interface. The > examples all seem to just do a transform, then transform back to confirm > the same function comes out again; that works fine over here, but I can't > seem to figure out how the actual fourier transform itself is stored so it > can be manipulated or used before transforming back. I wrote a short test > program: > http://pastebin.com/4saWgqLu > It fills a vector with a sine, then transforms it and displays the result, > which looks like this: > http://imageshack.us/a/img37/1135/whatph.png > (the original vector is 2000 points; I expect this is 4000 because it > somehow represents a complex vector?) > I would expect a smooth function with two peaks on the far left and right, > but instead all it shows is a repetitively garbled mess in the first half > of the output and mostly zeroes in the second. Am I just doing something > wrong? If so, what, and if not, what exactly does this output represent? > How does it relate to the output I'm expecting? Any help would be greatly > appreciated. > You want something this is a nice multiple of the fundamental of the box, so k pi i/N. Then you will get a spike. You can find out which frequencies are where in the FFTW docs. Matt > Yours, > > Leon Buikstra -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jzhong at scsolutions.com Thu Oct 25 17:10:52 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 25 Oct 2012 22:10:52 +0000 Subject: [petsc-users] PT-SCOTCH not available Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> Dear PETSc folks, I am trying to install PT-SCOTCH as the external package for PETSc. I used --with-ptscotch=1 --download-ptscotch=yes to install PETSc. No complains. But when I tried to use PT-SCOTCH, here is the message I got ... ZMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** PT-SCOTCH not available. Aborting. ... Any idea to make it work? Thanks, Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Oct 25 17:49:18 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Oct 2012 17:49:18 -0500 (CDT) Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> Message-ID: do you also have parmetis installed? Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Dear PETSc folks, > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > --with-ptscotch=1 > --download-ptscotch=yes > > to install PETSc. No complains. > > But when I tried to use PT-SCOTCH, here is the message I got > ... > ZMUMPS 4.10.0 > L U Solver for unsymmetric matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > PT-SCOTCH not available. > Aborting. > ... > > Any idea to make it work? > > Thanks, > > Jinquan > From jzhong at scsolutions.com Thu Oct 25 17:50:13 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 25 Oct 2012 22:50:13 +0000 Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> Yes, I used ./configure \ --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack.a \ --download-blacs --download-scalapack \ --with-mpi-dir=/data/Users/jzhong/mpich2 \ --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ --with-scalar-type=complex --with-precision=double --with-clanguage=c --with-fortran-kernels=generic \ --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps --download-superlu_dist \ --download-parmetis --download-metis --with-fortran-interfaces Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, October 25, 2012 3:49 PM To: PETSc users list Subject: Re: [petsc-users] PT-SCOTCH not available do you also have parmetis installed? Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Dear PETSc folks, > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > --with-ptscotch=1 > --download-ptscotch=yes > > to install PETSc. No complains. > > But when I tried to use PT-SCOTCH, here is the message I got ... > ZMUMPS 4.10.0 > L U Solver for unsymmetric matrices > Type of parallelism: Working host > > ****** ANALYSIS STEP ******** > > PT-SCOTCH not available. > Aborting. > ... > > Any idea to make it work? > > Thanks, > > Jinquan > From jzhong at scsolutions.com Thu Oct 25 18:50:28 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Thu, 25 Oct 2012 23:50:28 +0000 Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A3C@exchange.scsolutions.com> Here is what I did: 1. First I used ./configure \ --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack.a \ --download-blacs --download-scalapack \ --with-mpi-dir=/data/Users/jzhong/mpich2 \ --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ --with-scalar-type=complex --with-precision=double --with-clanguage=c --with-fortran-kernels=generic \ --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps --download-superlu_dist \ --download-parmetis --download-metis --with-fortran-interfaces And make ... to download and compile ptscotch during the first round of PETSc installation in /data/Users/jzhong/petsc-3.3-p3-option4p2; 2. Next I used ./configure \ --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack.a \ --download-blacs --download-scalapack \ --with-mpi-dir=/data/Users/jzhong/mpich2 \ --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ --with-scalar-type=complex --with-precision=double --with-clanguage=c --with-fortran-kernels=generic \ --with-ptscotch=1 \ --with-ptscotch-include=/data/Users/jzhong/petsc-3.3-p3-option4p2/externalpackages/scotch_5.1.12b_esmumps-p2/include \ --with-ptscotch-lib="/data/Users/jzhong/petsc-3.3-p3-option4p2/externalpackages/scotch_5.1.12b_esmumps-p2/lib/libptscotch.a /data/Users/jzhong/petsc-3.3-p3-option4p2/externalpackages/scotch_5.1.12b_esmumps-p2/lib/libptscotcherr.a /data/Users/jzhong/petsc-3.3-p3-option4p2/externalpackages/scotch_5.1.12b_esmumps-p2/lib/libptscotcherrexit.a /data/Users/jzhong/petsc-3.3-p3-option4p2/externalpackages/scotch_5.1.12b_esmumps-p2/lib/libptscotchparmetis.a" \ --download-mumps --download-superlu_dist \ --download-parmetis --download-metis --with-fortran-interfaces and make... to reconfigure PETSc the compatible object and include files for ptscotch during the second round of PETSc installation in a different folder /data/Users/jzhong/petsc-3.3-p3-option4p3. No complains 3. Compile and run my source code with the second build of PETSc. Here is the error info when I tried to use ptscotch instead of ParMETIS ... ZMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** PT-SCOTCH not available. Aborting. ... 4. I believe it got stuck in zmumps_part5.F #if ! defined(ptscotch) IF ((id%KEEP(244) .EQ. 2) .AND. (id%KEEP(245) .EQ. 1)) THEN id%INFO(1) = -38 IF(id%MYID .EQ.0 ) THEN WRITE(LP,'("PT-SCOTCH not available.")') WRITE(LP,'("Aborting.")') RETURN END IF END IF #endif This indicates in both rounds of installation of PETSc, PETSc did't inform mumps that defined(ptscotch) is TRUE. This may indicate a bug in ./configure. Jinquan From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jinquan Zhong Sent: Thursday, October 25, 2012 3:11 PM To: petsc-users at mcs.anl.gov Subject: [petsc-users] PT-SCOTCH not available Dear PETSc folks, I am trying to install PT-SCOTCH as the external package for PETSc. I used --with-ptscotch=1 --download-ptscotch=yes to install PETSc. No complains. But when I tried to use PT-SCOTCH, here is the message I got ... ZMUMPS 4.10.0 L U Solver for unsymmetric matrices Type of parallelism: Working host ****** ANALYSIS STEP ******** PT-SCOTCH not available. Aborting. ... Any idea to make it work? Thanks, Jinquan -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Oct 25 19:14:20 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Oct 2012 19:14:20 -0500 (CDT) Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> Message-ID: try using the attached patch and see if it works. patch -Np1 < mumps-ptscotch.py rm -rf externalpackages/scotch* ./configure .... Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Yes, I used > > ./configure \ > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack.a \ > --download-blacs --download-scalapack \ > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > --with-scalar-type=complex --with-precision=double --with-clanguage=c --with-fortran-kernels=generic \ > --with-ptscotch=1 --download-ptscotch=yes \ > --download-mumps --download-superlu_dist \ > --download-parmetis --download-metis --with-fortran-interfaces > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 3:49 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > do you also have parmetis installed? > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Dear PETSc folks, > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > --with-ptscotch=1 > > --download-ptscotch=yes > > > > to install PETSc. No complains. > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > ZMUMPS 4.10.0 > > L U Solver for unsymmetric matrices > > Type of parallelism: Working host > > > > ****** ANALYSIS STEP ******** > > > > PT-SCOTCH not available. > > Aborting. > > ... > > > > Any idea to make it work? > > > > Thanks, > > > > Jinquan > > > > -------------- next part -------------- diff --git a/config/PETSc/packages/MUMPS.py b/config/PETSc/packages/MUMPS.py --- a/config/PETSc/packages/MUMPS.py +++ b/config/PETSc/packages/MUMPS.py @@ -69,7 +69,7 @@ g.write('LMETIS = '+self.libraries.toString(self.parmetis.lib)+'\n') orderingsc += ' -Dmetis -Dparmetis' orderingsf += ' '+self.compilers.FortranDefineCompilerOption+'metis '+self.compilers.FortranDefineCompilerOption+'parmetis' - elif self.ptscotch.found: + if self.ptscotch.found: g.write('ISCOTCH = '+self.headers.toString(self.ptscotch.include)+'\n') g.write('LSCOTCH = '+self.libraries.toString(self.ptscotch.lib)+'\n') orderingsc += ' -Dscotch -Dptscotch' @@ -139,7 +139,7 @@ if not self.framework.argDB['with-mumps-serial']: if self.parmetis.found: self.deps.append(self.parmetis) - elif self.ptscotch.found: + if self.ptscotch.found: self.deps.append(self.ptscotch) PETSc.package.NewPackage.configureLibrary(self) # [parallem mumps] make sure either ptscotch or parmetis is enabled diff --git a/config/PETSc/packages/PTScotch.py b/config/PETSc/packages/PTScotch.py --- a/config/PETSc/packages/PTScotch.py +++ b/config/PETSc/packages/PTScotch.py @@ -4,9 +4,9 @@ def __init__(self, framework): PETSc.package.NewPackage.__init__(self, framework) #'https://gforge.inria.fr/frs/download.php/28978/scotch_5.1.12b_esmumps.tar.gz' - self.download = ['http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch_5.1.12b_esmumps-p2.tar.gz'] + self.download = ['http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch_5.1.12b_esmumps-p1.tar.gz'] self.downloadfilename = 'scotch' - self.liblist = [['libptscotch.a','libptscotcherr.a']] + self.liblist = [['libptesmumps.a', 'libptscotch.a','libptscotcherr.a']] self.functions = ['SCOTCH_archBuild'] self.includes = ['ptscotch.h'] self.requires32bitint = 0 From jzhong at scsolutions.com Thu Oct 25 19:26:23 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Fri, 26 Oct 2012 00:26:23 +0000 Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A50@exchange.scsolutions.com> Did you mean patch -Np1 < mumps-ptscotch.patch ? -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, October 25, 2012 5:14 PM To: PETSc users list Subject: Re: [petsc-users] PT-SCOTCH not available try using the attached patch and see if it works. patch -Np1 < mumps-ptscotch.py rm -rf externalpackages/scotch* ./configure .... Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Yes, I used > > ./configure \ > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack. > a \ --download-blacs --download-scalapack \ > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > --with-scalar-type=complex --with-precision=double --with-clanguage=c > --with-fortran-kernels=generic \ > --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps > --download-superlu_dist \ --download-parmetis --download-metis > --with-fortran-interfaces > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 3:49 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > do you also have parmetis installed? > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Dear PETSc folks, > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > --with-ptscotch=1 > > --download-ptscotch=yes > > > > to install PETSc. No complains. > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > ZMUMPS 4.10.0 > > L U Solver for unsymmetric matrices > > Type of parallelism: Working host > > > > ****** ANALYSIS STEP ******** > > > > PT-SCOTCH not available. > > Aborting. > > ... > > > > Any idea to make it work? > > > > Thanks, > > > > Jinquan > > > > From balay at mcs.anl.gov Thu Oct 25 19:27:07 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Oct 2012 19:27:07 -0500 (CDT) Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A50@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A50@exchange.scsolutions.com> Message-ID: yeah. sorry about the typo.. satish On Fri, 26 Oct 2012, Jinquan Zhong wrote: > Did you mean > > patch -Np1 < mumps-ptscotch.patch > > ? > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 5:14 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > try using the attached patch and see if it works. > > patch -Np1 < mumps-ptscotch.py > rm -rf externalpackages/scotch* > ./configure .... > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Yes, I used > > > > ./configure \ > > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack. > > a \ --download-blacs --download-scalapack \ > > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > > --with-scalar-type=complex --with-precision=double --with-clanguage=c > > --with-fortran-kernels=generic \ > > --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps > > --download-superlu_dist \ --download-parmetis --download-metis > > --with-fortran-interfaces > > > > Jinquan > > > > > > -----Original Message----- > > From: petsc-users-bounces at mcs.anl.gov > > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > > Sent: Thursday, October 25, 2012 3:49 PM > > To: PETSc users list > > Subject: Re: [petsc-users] PT-SCOTCH not available > > > > do you also have parmetis installed? > > > > Satish > > > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > > > Dear PETSc folks, > > > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > > --with-ptscotch=1 > > > --download-ptscotch=yes > > > > > > to install PETSc. No complains. > > > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > > ZMUMPS 4.10.0 > > > L U Solver for unsymmetric matrices > > > Type of parallelism: Working host > > > > > > ****** ANALYSIS STEP ******** > > > > > > PT-SCOTCH not available. > > > Aborting. > > > ... > > > > > > Any idea to make it work? > > > > > > Thanks, > > > > > > Jinquan > > > > > > > > From jzhong at scsolutions.com Thu Oct 25 19:30:30 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Fri, 26 Oct 2012 00:30:30 +0000 Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A5C@exchange.scsolutions.com> I got this =============================================================================== TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:133) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Downloaded ptscotch could not be used. Please check install in /data/Users/jzhong/petsc-3.3-p3-option4p2/linux-gnu-c-debug Not quite working yet. Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, October 25, 2012 5:14 PM To: PETSc users list Subject: Re: [petsc-users] PT-SCOTCH not available try using the attached patch and see if it works. patch -Np1 < mumps-ptscotch.py rm -rf externalpackages/scotch* ./configure .... Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Yes, I used > > ./configure \ > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack. > a \ --download-blacs --download-scalapack \ > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > --with-scalar-type=complex --with-precision=double --with-clanguage=c > --with-fortran-kernels=generic \ > --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps > --download-superlu_dist \ --download-parmetis --download-metis > --with-fortran-interfaces > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 3:49 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > do you also have parmetis installed? > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Dear PETSc folks, > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > --with-ptscotch=1 > > --download-ptscotch=yes > > > > to install PETSc. No complains. > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > ZMUMPS 4.10.0 > > L U Solver for unsymmetric matrices > > Type of parallelism: Working host > > > > ****** ANALYSIS STEP ******** > > > > PT-SCOTCH not available. > > Aborting. > > ... > > > > Any idea to make it work? > > > > Thanks, > > > > Jinquan > > > > From jzhong at scsolutions.com Thu Oct 25 19:32:25 2012 From: jzhong at scsolutions.com (Jinquan Zhong) Date: Fri, 26 Oct 2012 00:32:25 +0000 Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A5C@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A5C@exchange.scsolutions.com> Message-ID: <7237384C7A8F2642A8854B751619FA31704F0A69@exchange.scsolutions.com> The log file says: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Downloaded ptscotch could not be used. Please check install in /data/Users/jzhong/petsc-3.3-p3-option4p2/linux-gnu-c-debug ******************************************************************************* File "./config/configure.py", line 311, in petsc_configure framework.configure(out = sys.stdout) File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/framework.py", line 933, in configure child.configure() File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 526, in configure self.executeTest(self.configureLibrary) File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/base.py", line 115, in executeTest ret = apply(test, args,kargs) File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 453, in configureLibrary for location, directory, lib, incl in self.generateGuesses(): File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 235, in generateGuesses raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jinquan Zhong Sent: Thursday, October 25, 2012 5:31 PM To: 'PETSc users list' Subject: Re: [petsc-users] PT-SCOTCH not available I got this =============================================================================== TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:133) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Downloaded ptscotch could not be used. Please check install in /data/Users/jzhong/petsc-3.3-p3-option4p2/linux-gnu-c-debug Not quite working yet. Jinquan -----Original Message----- From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay Sent: Thursday, October 25, 2012 5:14 PM To: PETSc users list Subject: Re: [petsc-users] PT-SCOTCH not available try using the attached patch and see if it works. patch -Np1 < mumps-ptscotch.py rm -rf externalpackages/scotch* ./configure .... Satish On Thu, 25 Oct 2012, Jinquan Zhong wrote: > Yes, I used > > ./configure \ > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack. > a \ --download-blacs --download-scalapack \ > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > --with-scalar-type=complex --with-precision=double --with-clanguage=c > --with-fortran-kernels=generic \ > --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps > --download-superlu_dist \ --download-parmetis --download-metis > --with-fortran-interfaces > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 3:49 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > do you also have parmetis installed? > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Dear PETSc folks, > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > --with-ptscotch=1 > > --download-ptscotch=yes > > > > to install PETSc. No complains. > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > ZMUMPS 4.10.0 > > L U Solver for unsymmetric matrices > > Type of parallelism: Working host > > > > ****** ANALYSIS STEP ******** > > > > PT-SCOTCH not available. > > Aborting. > > ... > > > > Any idea to make it work? > > > > Thanks, > > > > Jinquan > > > > From balay at mcs.anl.gov Thu Oct 25 19:40:41 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 25 Oct 2012 19:40:41 -0500 (CDT) Subject: [petsc-users] PT-SCOTCH not available In-Reply-To: <7237384C7A8F2642A8854B751619FA31704F0A69@exchange.scsolutions.com> References: <7237384C7A8F2642A8854B751619FA31704F0A1E@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A2F@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A5C@exchange.scsolutions.com> <7237384C7A8F2642A8854B751619FA31704F0A69@exchange.scsolutions.com> Message-ID: do 'rm -rf externalpackages linux-gnu-c-debug' and retry configure. If you still have issues - send configure.log to petsc-maint Satish On Fri, 26 Oct 2012, Jinquan Zhong wrote: > The log file says: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Downloaded ptscotch could not be used. Please check install in /data/Users/jzhong/petsc-3.3-p3-option4p2/linux-gnu-c-debug > ******************************************************************************* > > File "./config/configure.py", line 311, in petsc_configure > framework.configure(out = sys.stdout) > File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/framework.py", line 933, in configure > child.configure() > File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 526, in configure > self.executeTest(self.configureLibrary) > File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/base.py", line 115, in executeTest > ret = apply(test, args,kargs) > File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 453, in configureLibrary > for location, directory, lib, incl in self.generateGuesses(): > File "/data/Users/jzhong/petsc-3.3-p3-option4p2/config/BuildSystem/config/package.py", line 235, in generateGuesses > raise RuntimeError('Downloaded '+self.package+' could not be used. Please check install in '+d+'\n') > > > > Jinquan > > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jinquan Zhong > Sent: Thursday, October 25, 2012 5:31 PM > To: 'PETSc users list' > Subject: Re: [petsc-users] PT-SCOTCH not available > > I got this > > > =============================================================================== > TESTING: check from config.libraries(config/BuildSystem/config/libraries.py:133) > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Downloaded ptscotch could not be used. Please check install in /data/Users/jzhong/petsc-3.3-p3-option4p2/linux-gnu-c-debug > > Not quite working yet. > > Jinquan > > > -----Original Message----- > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > Sent: Thursday, October 25, 2012 5:14 PM > To: PETSc users list > Subject: Re: [petsc-users] PT-SCOTCH not available > > try using the attached patch and see if it works. > > patch -Np1 < mumps-ptscotch.py > rm -rf externalpackages/scotch* > ./configure .... > > Satish > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > Yes, I used > > > > ./configure \ > > --with-blas-lib=/data/Users/jzhong/scalapack-2.0.1/lib/librefblas.a \ > > --with-lapack-lib=/data/Users/jzhong/scalapack-2.0.1/lib/libreflapack. > > a \ --download-blacs --download-scalapack \ > > --with-mpi-dir=/data/Users/jzhong/mpich2 \ > > --with-mpiexec=/data/Users/jzhong/mpich2/bin/mpiexec \ > > --with-cmake=/data/Users/jzhong/CMAKE/cmake-2.8.8/bin/cmake \ > > --with-scalar-type=complex --with-precision=double --with-clanguage=c > > --with-fortran-kernels=generic \ > > --with-ptscotch=1 --download-ptscotch=yes \ --download-mumps > > --download-superlu_dist \ --download-parmetis --download-metis > > --with-fortran-interfaces > > > > Jinquan > > > > > > -----Original Message----- > > From: petsc-users-bounces at mcs.anl.gov > > [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Satish Balay > > Sent: Thursday, October 25, 2012 3:49 PM > > To: PETSc users list > > Subject: Re: [petsc-users] PT-SCOTCH not available > > > > do you also have parmetis installed? > > > > Satish > > > > On Thu, 25 Oct 2012, Jinquan Zhong wrote: > > > > > Dear PETSc folks, > > > > > > I am trying to install PT-SCOTCH as the external package for PETSc. I used > > > --with-ptscotch=1 > > > --download-ptscotch=yes > > > > > > to install PETSc. No complains. > > > > > > But when I tried to use PT-SCOTCH, here is the message I got ... > > > ZMUMPS 4.10.0 > > > L U Solver for unsymmetric matrices > > > Type of parallelism: Working host > > > > > > ****** ANALYSIS STEP ******** > > > > > > PT-SCOTCH not available. > > > Aborting. > > > ... > > > > > > Any idea to make it work? > > > > > > Thanks, > > > > > > Jinquan > > > > > > > > From thomas.witkowski at tu-dresden.de Fri Oct 26 04:20:19 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Fri, 26 Oct 2012 11:20:19 +0200 Subject: [petsc-users] Replace schur solver in fieldsplit by a matrix multiplication Message-ID: <508A55D3.6000200@tu-dresden.de> Is there a simple way to replace the Schur ksp in fieldsplit preconditioner by a matrix multiplication? So, instead of solving with some Schur operator approximation I just want to make a matrix-vector multiplication, where the matrix is a sparse approximation of the inverse of the Schur complement. Thanks for any suggestions! Thomas From thomas.witkowski at tu-dresden.de Fri Oct 26 04:23:31 2012 From: thomas.witkowski at tu-dresden.de (Thomas Witkowski) Date: Fri, 26 Oct 2012 11:23:31 +0200 Subject: [petsc-users] Replace schur solver in fieldsplit by a matrix multiplication In-Reply-To: <508A55D3.6000200@tu-dresden.de> References: <508A55D3.6000200@tu-dresden.de> Message-ID: <508A5693.3030604@tu-dresden.de> Just to add what I have now is to set the KSP type to PREONLY and the preconditioner is a PCSHELL which makes this matrix vector multiplication. But is there is simpler way? Thomas Am 26.10.2012 11:20, schrieb Thomas Witkowski: > Is there a simple way to replace the Schur ksp in fieldsplit > preconditioner by a matrix multiplication? So, instead of solving with > some Schur operator approximation I just want to make a matrix-vector > multiplication, where the matrix is a sparse approximation of the > inverse of the Schur complement. Thanks for any suggestions! > > Thomas From jedbrown at mcs.anl.gov Fri Oct 26 05:53:05 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Fri, 26 Oct 2012 03:53:05 -0700 Subject: [petsc-users] Replace schur solver in fieldsplit by a matrix multiplication In-Reply-To: <508A5693.3030604@tu-dresden.de> References: <508A55D3.6000200@tu-dresden.de> <508A5693.3030604@tu-dresden.de> Message-ID: On Fri, Oct 26, 2012 at 2:23 AM, Thomas Witkowski < thomas.witkowski at tu-dresden.de> wrote: > Just to add what I have now is to set the KSP type to PREONLY and the > preconditioner is a PCSHELL which makes this matrix vector multiplication. > But is there is simpler way? > The PCSHELL could be a PCMAT (or PCSPAI), but I think this is the consistent approach. We do the same PREONLY approach to do an approximate solve on subdomains of ASM, for example. > > Thomas > > Am 26.10.2012 11:20, schrieb Thomas Witkowski: > > Is there a simple way to replace the Schur ksp in fieldsplit >> preconditioner by a matrix multiplication? So, instead of solving with some >> Schur operator approximation I just want to make a matrix-vector >> multiplication, where the matrix is a sparse approximation of the inverse >> of the Schur complement. Thanks for any suggestions! >> >> Thomas >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Oct 26 11:04:25 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 26 Oct 2012 17:04:25 +0100 (BST) Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! Message-ID: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> Dear Developers and members, I intend to solve system Ax=b with two preconditioners as follows: P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b where P1 and P2 are exclusively available as matrices. This might be crazy to many. I thought to start with PCCOMPOSITE but i do not think it does exactly what I intend.? Any Idea would be appreciated. A beginner!!! Abdul -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Oct 26 11:22:29 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Oct 2012 12:22:29 -0400 Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! In-Reply-To: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> References: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 26, 2012 at 12:04 PM, Abdul Hanan Sheikh wrote: > > Dear Developers and members, > > I intend to solve system Ax=b with two preconditioners as follows: > > P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b > I assume you mean P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b With this setup, I think its easiest to use PCKSP with the matrix for the KSP being P2. That would give you P2^{-1} A x = P2^{-1} b Now you can precondition that solve with another PCKSP with the matrix for that KSP being P1, and get what you want. This seems like a strange thing since you could just merge those matrices. Matt > where P1 and P2 are exclusively available as matrices. > This might be crazy to many. > > I thought to start with PCCOMPOSITE but i do not think it does exactly > what I intend. > > Any Idea would be appreciated. > > A beginner!!! > Abdul > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Oct 26 11:45:25 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 26 Oct 2012 17:45:25 +0100 (BST) Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! In-Reply-To: References: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> Message-ID: <1351269925.26820.YahooMailNeo@web171605.mail.ir2.yahoo.com> Thank you Matt, your assumption is alright. It was typo. I got this idea , yet if I want to invert P2 exactly by PCLU then how should I do ?? This is indeed, but I avoid to merge(matrix product) for some reasons. Thanks again. Abdul >________________________________ > From: Matthew Knepley >To: Abdul Hanan Sheikh ; PETSc users list >Sent: Friday, 26 October 2012, 18:22 >Subject: Re: [petsc-users] Two PCs on same residual in multiplicative fashion ! > > >On Fri, Oct 26, 2012 at 12:04 PM, Abdul Hanan Sheikh wrote: > > >> >>Dear Developers and members, >> >> >> >>I intend to solve system Ax=b with two preconditioners as follows: >> >> >> >>P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b > > >I assume you mean P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b > > > > >With this setup, I think its easiest to use PCKSP with the matrix for >the KSP being P2. That would give you > > >? P2^{-1} A x = P2^{-1} b > > >Now you can precondition that solve with another PCKSP with the matrix >for that KSP being P1, and get what you want. > > >This seems like a strange thing since you could just merge those matrices. > > >? ?Matt >? >where P1 and P2 are exclusively available as matrices. >> >>This might be crazy to many. >> >> >> >>I thought to start with PCCOMPOSITE but i do not think it does exactly what I intend.? >> >> >>Any Idea would be appreciated. >> >> >> >>A beginner!!! >>Abdul >> > > > >-- >What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >-- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Oct 26 11:56:34 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Oct 2012 12:56:34 -0400 Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! In-Reply-To: <1351269925.26820.YahooMailNeo@web171605.mail.ir2.yahoo.com> References: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> <1351269925.26820.YahooMailNeo@web171605.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 26, 2012 at 12:45 PM, Abdul Hanan Sheikh wrote: > Thank you Matt, your assumption is alright. It was typo. > > I got this idea , yet if I want to invert P2 exactly by PCLU then how > should I do ?? > Read this: http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCKSP.html This makes things more complicated. You can use PCOMPOSITE with multiplicative, but its not exactly what you have written. This is never used because its not really effective, and doesn't make much sense from a theory perspective. Are you sure you want to do this? Matt This is indeed, but I avoid to merge(matrix product) for some reasons. > > Thanks again. > Abdul > > ------------------------------ > *From:* Matthew Knepley > *To:* Abdul Hanan Sheikh ; PETSc users list < > petsc-users at mcs.anl.gov> > *Sent:* Friday, 26 October 2012, 18:22 > *Subject:* Re: [petsc-users] Two PCs on same residual in multiplicative > fashion ! > > On Fri, Oct 26, 2012 at 12:04 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > > Dear Developers and members, > > I intend to solve system Ax=b with two preconditioners as follows: > > P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b > > > I assume you mean P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b > > > > With this setup, I think its easiest to use PCKSP with the matrix for > the KSP being P2. That would give you > > P2^{-1} A x = P2^{-1} b > > Now you can precondition that solve with another PCKSP with the matrix > for that KSP being P1, and get what you want. > > This seems like a strange thing since you could just merge those matrices. > > Matt > > > where P1 and P2 are exclusively available as matrices. > This might be crazy to many. > > I thought to start with PCCOMPOSITE but i do not think it does exactly > what I intend. > > Any Idea would be appreciated. > > A beginner!!! > Abdul > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Oct 26 12:05:35 2012 From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh) Date: Fri, 26 Oct 2012 18:05:35 +0100 (BST) Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! In-Reply-To: References: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> <1351269925.26820.YahooMailNeo@web171605.mail.ir2.yahoo.com> Message-ID: <1351271135.14319.YahooMailNeo@web171604.mail.ir2.yahoo.com> Yes, it sounds crazy when we look exclusively at this. But I do not do this exclusively. Basically,? this happening [ P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b ] occurs as a part of my two-level solver. Any how I have to do this. I had a realization, PCCOMPOSITE does not do exactly what I want. I read about PCCOMPOSITE in manual. :( Thanks, Abdul >________________________________ > From: Matthew Knepley >To: Abdul Hanan Sheikh ; PETSc users list >Sent: Friday, 26 October 2012, 18:56 >Subject: Re: [petsc-users] Two PCs on same residual in multiplicative fashion ! > > >On Fri, Oct 26, 2012 at 12:45 PM, Abdul Hanan Sheikh wrote: > >Thank you Matt, your assumption is alright. It was typo. >> >>I got this idea , yet if I want to invert P2 exactly by PCLU then how should I do ??? > > >Read this:?http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCKSP.html > > >This makes things more complicated. You can use PCOMPOSITE with multiplicative, but its not >exactly what you have written. This is never used because its not really effective, and doesn't make >much sense from a theory perspective. Are you sure you want to do this? > > >? ? Matt > > >This is indeed, but I avoid to merge(matrix product) for some reasons. >> >> >> >>Thanks again. >> >>Abdul >> >> >> >> >>>________________________________ >>> From: Matthew Knepley >>>To: Abdul Hanan Sheikh ; PETSc users list >>>Sent: Friday, 26 October 2012, 18:22 >>>Subject: Re: [petsc-users] Two PCs on same residual in multiplicative fashion ! >>> >>> >>>On Fri, Oct 26, 2012 at 12:04 PM, Abdul Hanan Sheikh wrote: >>> >>> >>>> >>>>Dear Developers and members, >>>> >>>> >>>> >>>>I intend to solve system Ax=b with two preconditioners as follows: >>>> >>>> >>>> >>>>P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b >>> >>> >>>I assume you mean P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b >>> >>> >>> >>> >>>With this setup, I think its easiest to use PCKSP with the matrix for >>>the KSP being P2. That would give you >>> >>> >>>? P2^{-1} A x = P2^{-1} b >>> >>> >>>Now you can precondition that solve with another PCKSP with the matrix >>>for that KSP being P1, and get what you want. >>> >>> >>>This seems like a strange thing since you could just merge those matrices. >>> >>> >>>? ?Matt >>>? >>>where P1 and P2 are exclusively available as matrices. >>>> >>>>This might be crazy to many. >>>> >>>> >>>> >>>>I thought to start with PCCOMPOSITE but i do not think it does exactly what I intend.? >>>> >>>> >>>>Any Idea would be appreciated. >>>> >>>> >>>> >>>>A beginner!!! >>>>Abdul >>>> >>> >>> >>>-- >>>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>-- Norbert Wiener >>> >>> > > >? > > > >-- >What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >-- Norbert Wiener > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Oct 26 12:13:48 2012 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Oct 2012 13:13:48 -0400 Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! In-Reply-To: <1351271135.14319.YahooMailNeo@web171604.mail.ir2.yahoo.com> References: <1351267465.26815.YahooMailNeo@web171603.mail.ir2.yahoo.com> <1351269925.26820.YahooMailNeo@web171605.mail.ir2.yahoo.com> <1351271135.14319.YahooMailNeo@web171604.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 26, 2012 at 1:05 PM, Abdul Hanan Sheikh wrote: > > Yes, it sounds crazy when we look exclusively at this. But I do not do > this exclusively. > Basically, this happening [ P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b ] > occurs as a part of my two-level solver. > Any how I have to do this. > > I had a realization, PCCOMPOSITE does not do exactly what I want. I read > about PCCOMPOSITE in manual. :( > Just use PCSHELL, and in the PCApply() function do two KSPSolve()s, one with each matrix. Matt > Thanks, > Abdul > > > ------------------------------ > *From:* Matthew Knepley > *To:* Abdul Hanan Sheikh ; PETSc users list < > petsc-users at mcs.anl.gov> > *Sent:* Friday, 26 October 2012, 18:56 > *Subject:* Re: [petsc-users] Two PCs on same residual in multiplicative > fashion ! > > On Fri, Oct 26, 2012 at 12:45 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > Thank you Matt, your assumption is alright. It was typo. > > I got this idea , yet if I want to invert P2 exactly by PCLU then how > should I do ?? > > > Read this: > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCKSP.html > > This makes things more complicated. You can use PCOMPOSITE with > multiplicative, but its not > exactly what you have written. This is never used because its not really > effective, and doesn't make > much sense from a theory perspective. Are you sure you want to do this? > > Matt > > This is indeed, but I avoid to merge(matrix product) for some reasons. > > Thanks again. > Abdul > > ------------------------------ > *From:* Matthew Knepley > *To:* Abdul Hanan Sheikh ; PETSc users list < > petsc-users at mcs.anl.gov> > *Sent:* Friday, 26 October 2012, 18:22 > *Subject:* Re: [petsc-users] Two PCs on same residual in multiplicative > fashion ! > > On Fri, Oct 26, 2012 at 12:04 PM, Abdul Hanan Sheikh < > hanangul12 at yahoo.co.uk> wrote: > > > Dear Developers and members, > > I intend to solve system Ax=b with two preconditioners as follows: > > P1^{-1} P2^{-1} A x = P^{-1} P^{-1} b > > > I assume you mean P1^{-1} P2^{-1} A x = P1^{-1} P2^{-1} b > > > > With this setup, I think its easiest to use PCKSP with the matrix for > the KSP being P2. That would give you > > P2^{-1} A x = P2^{-1} b > > Now you can precondition that solve with another PCKSP with the matrix > for that KSP being P1, and get what you want. > > This seems like a strange thing since you could just merge those matrices. > > Matt > > > where P1 and P2 are exclusively available as matrices. > This might be crazy to many. > > I thought to start with PCCOMPOSITE but i do not think it does exactly > what I intend. > > Any Idea would be appreciated. > > A beginner!!! > Abdul > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanangul12 at yahoo.co.uk Fri Oct 26 13:12:44 2012 From: hanangul12 at yahoo.co.uk (hanangul12) Date: Fri, 26 Oct 2012 18:12:44 GMT Subject: [petsc-users] Two PCs on same residual in multiplicative fashion ! Message-ID: <304518.11092.bm@smtp126.mail.ukl.yahoo.com> An HTML attachment was scrubbed... URL: From ckhuangf at gmail.com Sat Oct 27 15:46:25 2012 From: ckhuangf at gmail.com (Chung-Kan Huang) Date: Sat, 27 Oct 2012 15:46:25 -0500 Subject: [petsc-users] Expecting zeros in solution when A_{I, I} != 0 A_{I, J} = 0. where I!=J and B_I = 0. Message-ID: Hi, I am solving a linear system A * X = B that sometime contains A_{I,I} != 0 A_{I,J} = 0. where I!=J and B_I = 0. so I would expect X_i to be perfect zero not some tiny small value. The same linear system is solved by other solvers but they gave zero for aforementioned situation even the system is fair large to our applications. I have tried several configurations in PETSc hoping that problem can be solved but all attempts failed. I wonder if someone can give me some pointers that help to solve this issue. Thanks, Chung-Kan -- *Cheers* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at mcs.anl.gov Sat Oct 27 18:07:15 2012 From: bsmith at mcs.anl.gov (Barry Smith) Date: Sat, 27 Oct 2012 18:07:15 -0500 Subject: [petsc-users] Expecting zeros in solution when A_{I, I} != 0 A_{I, J} = 0. where I!=J and B_I = 0. In-Reply-To: References: Message-ID: PETSc, by default, using iterative solvers which only solve the problem within a certain accuracy. You can control this accuracy by using -ksp_rtol tol where for example you could pick tol = 1.0e-12 You can also use a direct solver -pc_type lu Run with -ksp_monitor_true_residual -ksp_converged_reason to see if the linear solver is solving to the requested accuracy. Barry On Oct 27, 2012, at 3:46 PM, Chung-Kan Huang wrote: > Hi, > > I am solving a linear system A * X = B that sometime contains A_{I,I} != 0 A_{I,J} = 0. where I!=J and B_I = 0. so I would expect X_i to be perfect zero not some tiny small value. The same linear system is solved by other solvers but they gave zero for aforementioned situation even the system is fair large to our applications. I have tried several configurations in PETSc hoping that problem can be solved but all attempts failed. I wonder if someone can give me some pointers that help to solve this issue. Thanks, > > Chung-Kan > > > > -- > Cheers > > From w_ang_temp at 163.com Sun Oct 28 08:08:57 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 28 Oct 2012 21:08:57 +0800 (CST) Subject: [petsc-users] How rto use multigrid? Message-ID: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> Hello, I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. So are there some typical examples or details about multigrid? Is it used just like other preconditioners like jacobi, sor, which can be simply used in the cammand line options? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Oct 28 08:17:00 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 28 Oct 2012 06:17:00 -0700 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> Message-ID: Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. What kind of problem are you solving? On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: > Hello, > I want to use the multigrid as a preconditioner. The introduction about > it in the manual is little. > So are there some typical examples or details about multigrid? Is it used > just like other preconditioners > like jacobi, sor, which can be simply used in the cammand line options? > Thanks. > Jim > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sun Oct 28 08:38:40 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 28 Oct 2012 21:38:40 +0800 (CST) Subject: [petsc-users] How rto use multigrid? In-Reply-To: References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> Message-ID: <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> Hello, Jed Thanks for your timely reply. I deal with the soil-water coupled problem in geotechnical engineering, whose stiffness matrix is ill-conditioned. I have did some work about it, mainly finding the effective solvers and preconditioners. I used the command line option like this: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason. So, I also want to use the multigrid just like the simple command. There is only a little introduction about multigrid in the manual. Multigrid is complex and not a easy thing for me, so I just need to konw how to use it simply in PETSc to solve the Ax=b system. Thanks. Jim >At 2012-10-28 21:17:00,"Jed Brown" wrote: >Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set >interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. >What kind of problem are you solving? >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: >>Hello, >> I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. >>So are there some typical examples or details about multigrid? Is it used just like other preconditioners >>like jacobi, sor, which can be simply used in the cammand line options? >> Thanks. >> Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Oct 28 08:46:23 2012 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 28 Oct 2012 09:46:23 -0400 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> Message-ID: On Sun, Oct 28, 2012 at 9:38 AM, w_ang_temp wrote: > Hello, Jed > Thanks for your timely reply. I deal with the soil-water coupled > problem in geotechnical engineering, > whose stiffness matrix is ill-conditioned. I have did some work about it, > mainly finding the effective > solvers and preconditioners. I used the command line option like this: > mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 > -ksp_converged_reason. > If SOR works as a preconditioner, then definitely use AMG as Jed suggested. It is almost certain to work. Matt > So, I also want to use the multigrid just like the simple command. > There is only a little introduction > about multigrid in the manual. Multigrid is complex and not a easy thing > for me, so I just need to konw how > to use it simply in PETSc to solve the Ax=b system. > Thanks. > Jim > > > > > > >At 2012-10-28 21:17:00,"Jed Brown" wrote: > > >Algebraic multigrid can be used directly, -pc_type gamg > -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG > interface to set >interpolation (and provide a coarse operator for > non-Galerkin) or use a DM that provides coarsening capability. > > >What kind of problem are you solving? > >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: > >> >>Hello, >> >> I want to use the multigrid as a preconditioner. The introduction >> about it in the manual is little. >> >>So are there some typical examples or details about multigrid? Is it >> used just like other preconditioners >> >>like jacobi, sor, which can be simply used in the cammand line options? >> >> Thanks. >> >> >> Jim >> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Sun Oct 28 08:48:44 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Sun, 28 Oct 2012 21:48:44 +0800 (CST) Subject: [petsc-users] How rto use multigrid? In-Reply-To: References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> Message-ID: <33e308f3.107c3.13aa7a1b6e5.Coremail.w_ang_temp@163.com> Thanks. I will have a try. Jim ? 2012-10-28 21:46:23?"Matthew Knepley" ??? On Sun, Oct 28, 2012 at 9:38 AM, w_ang_temp wrote: Hello, Jed Thanks for your timely reply. I deal with the soil-water coupled problem in geotechnical engineering, whose stiffness matrix is ill-conditioned. I have did some work about it, mainly finding the effective solvers and preconditioners. I used the command line option like this: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason. If SOR works as a preconditioner, then definitely use AMG as Jed suggested. It is almost certain to work. Matt So, I also want to use the multigrid just like the simple command. There is only a little introduction about multigrid in the manual. Multigrid is complex and not a easy thing for me, so I just need to konw how to use it simply in PETSc to solve the Ax=b system. Thanks. Jim >At 2012-10-28 21:17:00,"Jed Brown" wrote: >Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set >interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. >What kind of problem are you solving? >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: >>Hello, >> I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. >>So are there some typical examples or details about multigrid? Is it used just like other preconditioners >>like jacobi, sor, which can be simply used in the cammand line options? >> Thanks. >> Jim -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Sun Oct 28 09:25:31 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Sun, 28 Oct 2012 07:25:31 -0700 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> Message-ID: Try the simple option I just sent. On Oct 28, 2012 6:38 AM, "w_ang_temp" wrote: > Hello, Jed > Thanks for your timely reply. I deal with the soil-water coupled > problem in geotechnical engineering, > whose stiffness matrix is ill-conditioned. I have did some work about it, > mainly finding the effective > solvers and preconditioners. I used the command line option like this: > mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 > -ksp_converged_reason. > So, I also want to use the multigrid just like the simple command. > There is only a little introduction > about multigrid in the manual. Multigrid is complex and not a easy thing > for me, so I just need to konw how > to use it simply in PETSc to solve the Ax=b system. > Thanks. > Jim > > > > > > >At 2012-10-28 21:17:00,"Jed Brown" wrote: > > >Algebraic multigrid can be used directly, -pc_type gamg > -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG > interface to set >interpolation (and provide a coarse operator for > non-Galerkin) or use a DM that provides coarsening capability. > > >What kind of problem are you solving? > >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: > >> >>Hello, >> >> I want to use the multigrid as a preconditioner. The introduction >> about it in the manual is little. >> >>So are there some typical examples or details about multigrid? Is it >> used just like other preconditioners >> >>like jacobi, sor, which can be simply used in the cammand line options? >> >> Thanks. >> >> >> Jim >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Mon Oct 29 07:49:36 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 29 Oct 2012 20:49:36 +0800 (CST) Subject: [petsc-users] How rto use multigrid? In-Reply-To: References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> Message-ID: <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> Hello, Jed I use the command: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_rtol 1.0e-15 -ksp_converged_reason The error is as follows: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unable to find requested PC type gamg! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Mon Oct 29 05:40:13 2012 [0]PETSC ERROR: Libraries linked from /home/geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-opt/lib [0]PETSC ERROR: Configure run at Mon Jul 2 20:33:17 2012 [0]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 --with-debugging=0 --download-parmetis --download-mumps --download-scalapack --download-blacs [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCSetType() line 67 in src/ksp/pc/interface/pcset.c [0]PETSC ERROR: PCSetFromOptions() line 184 in src/ksp/pc/interface/pcset.c [0]PETSC ERROR: KSPSetFromOptions() line 286 in src/ksp/ksp/interface/itcl.c When I use 'mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason', it is ok. So what is the possible reason? Thanks. Jim At 2012-10-28 22:25:31,"Jed Brown" wrote: Try the simple option I just sent. On Oct 28, 2012 6:38 AM, "w_ang_temp" wrote: Hello, Jed Thanks for your timely reply. I deal with the soil-water coupled problem in geotechnical engineering, whose stiffness matrix is ill-conditioned. I have did some work about it, mainly finding the effective solvers and preconditioners. I used the command line option like this: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason. So, I also want to use the multigrid just like the simple command. There is only a little introduction about multigrid in the manual. Multigrid is complex and not a easy thing for me, so I just need to konw how to use it simply in PETSc to solve the Ax=b system. Thanks. Jim >At 2012-10-28 21:17:00,"Jed Brown" wrote: >Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set >interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. >What kind of problem are you solving? >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: >>Hello, >> I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. >>So are there some typical examples or details about multigrid? Is it used just like other preconditioners >>like jacobi, sor, which can be simply used in the cammand line options? >> Thanks. >> Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From agrayver at gfz-potsdam.de Mon Oct 29 08:02:24 2012 From: agrayver at gfz-potsdam.de (Alexander Grayver) Date: Mon, 29 Oct 2012 14:02:24 +0100 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> Message-ID: <508E7E60.2030301@gfz-potsdam.de> On 29.10.2012 13:49, w_ang_temp wrote: > Hello, Jed > I use the command: > mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type gamg -pc_gamg_agg_nsmooths > 1 -ksp_rtol 1.0e-15 -ksp_converged_reason > The error is as follows: > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing > external package needed for type > seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! > [0]PETSC ERROR: Unable to find requested PC type gamg! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > *[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 > 09:30:51 CDT 2012 * You should use petsc-3.3-p3. > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Mon Oct 29 > 05:40:13 2012 > [0]PETSC ERROR: Libraries linked from /home > /geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-opt/lib > [0]PETSC ERROR: Configure run at Mon Jul 2 20:33:17 2012 > [0]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 > --download-f-blas-lapack=1 --with-x=1 --with-debugging=0 > --download-parmetis --download-mumps --download-scalapack --download-blacs > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCSetType() line 67 in src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: PCSetFromOptions() line 184 in > src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: KSPSetFromOptions() line 286 in > src/ksp/ksp/interface/itcl.c > When I use 'mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol > 1.0e-15 -ksp_converged_reason', it is ok. So > what is the possible reason? > Thanks. > Jim > -- Regards, Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.adams at columbia.edu Mon Oct 29 08:57:49 2012 From: mark.adams at columbia.edu (Mark F. Adams) Date: Mon, 29 Oct 2012 09:57:49 -0400 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> Message-ID: <01255FF7-A444-48E8-AB48-AE76C710197C@columbia.edu> You need an updated PETSc. (I thought 3.2 had an early version of gamg ? but you need 3.3 or dev) Mark On Oct 29, 2012, at 8:49 AM, w_ang_temp wrote: > Hello, Jed > I use the command: > mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_rtol 1.0e-15 -ksp_converged_reason > The error is as follows: > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type > seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! > [0]PETSC ERROR: Unable to find requested PC type gamg! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Mon Oct 29 05:40:13 2012 > [0]PETSC ERROR: Libraries linked from /home /geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-opt/lib > [0]PETSC ERROR: Configure run at Mon Jul 2 20:33:17 2012 > [0]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 --with-debugging=0 --download-parmetis --download-mumps --download-scalapack --download-blacs > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: PCSetType() line 67 in src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: PCSetFromOptions() line 184 in src/ksp/pc/interface/pcset.c > [0]PETSC ERROR: KSPSetFromOptions() line 286 in src/ksp/ksp/interface/itcl.c > When I use 'mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason', it is ok. So > what is the possible reason? > Thanks. > Jim > > > > > > At 2012-10-28 22:25:31,"Jed Brown" wrote: > Try the simple option I just sent. > > On Oct 28, 2012 6:38 AM, "w_ang_temp" wrote: > Hello, Jed > Thanks for your timely reply. I deal with the soil-water coupled problem in geotechnical engineering, > whose stiffness matrix is ill-conditioned. I have did some work about it, mainly finding the effective > solvers and preconditioners. I used the command line option like this: > mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason. > So, I also want to use the multigrid just like the simple command. There is only a little introduction > about multigrid in the manual. Multigrid is complex and not a easy thing for me, so I just need to konw how > to use it simply in PETSc to solve the Ax=b system. > Thanks. > Jim > > > > > > >At 2012-10-28 21:17:00,"Jed Brown" wrote: > >Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set >interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. > > >What kind of problem are you solving? > > >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: > >>Hello, > >> I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. > >>So are there some typical examples or details about multigrid? Is it used just like other preconditioners > >>like jacobi, sor, which can be simply used in the cammand line options? > >> Thanks. > >> Jim > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Mon Oct 29 09:03:29 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Mon, 29 Oct 2012 22:03:29 +0800 (CST) Subject: [petsc-users] How rto use multigrid? In-Reply-To: <01255FF7-A444-48E8-AB48-AE76C710197C@columbia.edu> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <3fb03a6a.10633.13aa798827b.Coremail.w_ang_temp@163.com> <611bda2d.10ae0.13aac91f0d5.Coremail.w_ang_temp@163.com> <01255FF7-A444-48E8-AB48-AE76C710197C@columbia.edu> Message-ID: Thanks. I will have a try. Jim ? 2012-10-29 21:57:49?"Mark F. Adams" ??? You need an updated PETSc. (I thought 3.2 had an early version of gamg ? but you need 3.3 or dev) Mark On Oct 29, 2012, at 8:49 AM, w_ang_temp wrote: Hello, Jed I use the command: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_rtol 1.0e-15 -ksp_converged_reason The error is as follows: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type seehttp://www.mcs.anl.gov/petsc/petsc-as/documentation/installation.html#external! [0]PETSC ERROR: Unable to find requested PC type gamg! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex4f on a arch-linu named ubuntu by geo Mon Oct 29 05:40:13 2012 [0]PETSC ERROR: Libraries linked from /home /geo/soft/petsc/petsc-3.2-p7/arch-linux2-c-opt/lib [0]PETSC ERROR: Configure run at Mon Jul 2 20:33:17 2012 [0]PETSC ERROR: Configure options --with-mpi-dir=/home/geo/soft/mpich2 --download-f-blas-lapack=1 --with-x=1 --with-debugging=0 --download-parmetis --download-mumps --download-scalapack --download-blacs [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCSetType() line 67 in src/ksp/pc/interface/pcset.c [0]PETSC ERROR: PCSetFromOptions() line 184 in src/ksp/pc/interface/pcset.c [0]PETSC ERROR: KSPSetFromOptions() line 286 in src/ksp/ksp/interface/itcl.c When I use 'mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason', it is ok. So what is the possible reason? Thanks. Jim At 2012-10-28 22:25:31,"Jed Brown" wrote: Try the simple option I just sent. On Oct 28, 2012 6:38 AM, "w_ang_temp" wrote: Hello, Jed Thanks for your timely reply. I deal with the soil-water coupled problem in geotechnical engineering, whose stiffness matrix is ill-conditioned. I have did some work about it, mainly finding the effective solvers and preconditioners. I used the command line option like this: mpiexec -n 4 ./ex4f -ksp_type cgs -pc_type sor -ksp_rtol 1.0e-15 -ksp_converged_reason. So, I also want to use the multigrid just like the simple command. There is only a little introduction about multigrid in the manual. Multigrid is complex and not a easy thing for me, so I just need to konw how to use it simply in PETSc to solve the Ax=b system. Thanks. Jim >At 2012-10-28 21:17:00,"Jed Brown" wrote: >Algebraic multigrid can be used directly, -pc_type gamg -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG interface to set >interpolation (and provide a coarse operator for non-Galerkin) or use a DM that provides coarsening capability. >What kind of problem are you solving? >>On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: >>Hello, >> I want to use the multigrid as a preconditioner. The introduction about it in the manual is little. >>So are there some typical examples or details about multigrid? Is it used just like other preconditioners >>like jacobi, sor, which can be simply used in the cammand line options? >> Thanks. >> Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From erocha.ssa at gmail.com Mon Oct 29 14:51:29 2012 From: erocha.ssa at gmail.com (Eduardo) Date: Mon, 29 Oct 2012 17:51:29 -0200 Subject: [petsc-users] AOCreateBasic Message-ID: Hi, Can I use AOCreateBasic to create a mapping that has negative indexes? I mean, a mapping that starts at some -N and goes up to napp-1-N ? With no holes. Thanks in advance, Eduardo From Brian.Moore at sdstate.edu Mon Oct 29 15:29:19 2012 From: Brian.Moore at sdstate.edu (Moore, Brian (SDSU)) Date: Mon, 29 Oct 2012 20:29:19 +0000 Subject: [petsc-users] Trying to run ex11 (complex example) in $PETSC_DIR/src/ksp/ksp/examples/tutorials Message-ID: <24D6225D-52F7-4E63-A26A-CF7D71E00DFC@sdstate.edu> I am trying to make a build of petsc that will enable complex numbers. As per the manual in Sec. 13.7, I did a ./configure as: ./configure --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex and then tried to compile and make the ex11 example program from $PETSC_DIR/src/ksp/ksp/examples/tutorials The error I am seeing is listed below. Does anyone have an idea what the problem is? I was able to run several of the other example programs in this folder (ex1, ex2, ex23) and was able to run the fortran program ex11f. Thanks. -- Brian Moore, Research High Performance Computing Specialist SDSU UNSS SAD 119, Box 2201, Brookings, SD 57007-1598 605.688.4679 (wk) / 605.521.3062 (cell) moore at moore-pc2:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials> make runex11 1,8c1,20 < 0 KSP Residual norm 4.62271 < 1 KSP Residual norm 1.58711 < 2 KSP Residual norm 0.767563 < 3 KSP Residual norm 0.472102 < 4 KSP Residual norm 0.435655 < 5 KSP Residual norm 0.154866 < 6 KSP Residual norm < 1.e-11 < Norm of error < 1.e-12 iterations 6 --- > 1+1i > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatGetOwnershipRange()! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 4, Fri Oct 26 10:46:51 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex11 on a arch-linu named moore-pc2 by moore Mon Oct 29 15:27:59 2012 > [0]PETSC ERROR: Libraries linked from /home/moore/petsc-3.3-p4/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Mon Oct 29 15:00:29 2012 > [0]PETSC ERROR: Configure options --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOwnershipRange() line 5992 in /home/moore/petsc-3.3-p4/src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 83 in src/ksp/ksp/examples/tutorials/ex11.c > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 /home/moore/petsc-3.3-p4/src/ksp/ksp/examples/tutorials Possible problem with with ex11_1, diffs above \n========================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Brian.Moore at sdstate.edu Mon Oct 29 16:08:41 2012 From: Brian.Moore at sdstate.edu (Moore, Brian (SDSU)) Date: Mon, 29 Oct 2012 21:08:41 +0000 Subject: [petsc-users] Trying to run ex11 (complex example) in $PETSC_DIR/src/ksp/ksp/examples/tutorials In-Reply-To: <24D6225D-52F7-4E63-A26A-CF7D71E00DFC@sdstate.edu> References: <24D6225D-52F7-4E63-A26A-CF7D71E00DFC@sdstate.edu> Message-ID: Apologies--probably should have done this before I posted the question. I just downloaded and compiled the latest development version this afternoon and was successfully able to run this example. -- Brian Moore, Research High Performance Computing Specialist SDSU UNSS SAD 119, Box 2201, Brookings, SD 57007-1598 605.688.4679 (wk) / 605.521.3062 (cell) On Oct 29, 2012, at 3:29 PM, Moore, Brian (SDSU) wrote: I am trying to make a build of petsc that will enable complex numbers. As per the manual in Sec. 13.7, I did a ./configure as: ./configure --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex and then tried to compile and make the ex11 example program from $PETSC_DIR/src/ksp/ksp/examples/tutorials The error I am seeing is listed below. Does anyone have an idea what the problem is? I was able to run several of the other example programs in this folder (ex1, ex2, ex23) and was able to run the fortran program ex11f. Thanks. -- Brian Moore, Research High Performance Computing Specialist SDSU UNSS SAD 119, Box 2201, Brookings, SD 57007-1598 605.688.4679 (wk) / 605.521.3062 (cell) moore at moore-pc2:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials> make runex11 1,8c1,20 < 0 KSP Residual norm 4.62271 < 1 KSP Residual norm 1.58711 < 2 KSP Residual norm 0.767563 < 3 KSP Residual norm 0.472102 < 4 KSP Residual norm 0.435655 < 5 KSP Residual norm 0.154866 < 6 KSP Residual norm < 1.e-11 < Norm of error < 1.e-12 iterations 6 --- > 1+1i > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Object is in wrong state! > [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatGetOwnershipRange()! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 4, Fri Oct 26 10:46:51 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex11 on a arch-linu named moore-pc2 by moore Mon Oct 29 15:27:59 2012 > [0]PETSC ERROR: Libraries linked from /home/moore/petsc-3.3-p4/arch-linux2-c-debug/lib > [0]PETSC ERROR: Configure run at Mon Oct 29 15:00:29 2012 > [0]PETSC ERROR: Configure options --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatGetOwnershipRange() line 5992 in /home/moore/petsc-3.3-p4/src/mat/interface/matrix.c > [0]PETSC ERROR: main() line 83 in src/ksp/ksp/examples/tutorials/ex11.c > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 /home/moore/petsc-3.3-p4/src/ksp/ksp/examples/tutorials Possible problem with with ex11_1, diffs above \n========================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Oct 29 16:16:30 2012 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 29 Oct 2012 16:16:30 -0500 (CDT) Subject: [petsc-users] Trying to run ex11 (complex example) in $PETSC_DIR/src/ksp/ksp/examples/tutorials In-Reply-To: References: <24D6225D-52F7-4E63-A26A-CF7D71E00DFC@sdstate.edu> Message-ID: The attached patch [which is in petsc-dev] would fix the issue for petsc-3.3 aswell. cd petsc-3.3 patch -Np1 < ex11.diff Satish On Mon, 29 Oct 2012, Moore, Brian (SDSU) wrote: > > Apologies--probably should have done this before I posted the question. > I just downloaded and compiled the latest development version this afternoon and was successfully able to run this example. > > -- > Brian Moore, Research High Performance Computing Specialist > SDSU UNSS > SAD 119, Box 2201, Brookings, SD 57007-1598 > 605.688.4679 (wk) / 605.521.3062 (cell) > > > > > On Oct 29, 2012, at 3:29 PM, Moore, Brian (SDSU) wrote: > > > I am trying to make a build of petsc that will enable complex numbers. As per the manual in Sec. 13.7, I did a ./configure as: > > ./configure --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex > > and then tried to compile and make the ex11 example program from $PETSC_DIR/src/ksp/ksp/examples/tutorials > > The error I am seeing is listed below. Does anyone have an idea what the problem is? I was able to run several of the other example programs in this folder (ex1, ex2, ex23) and was able to run the fortran program ex11f. > Thanks. > > -- > Brian Moore, Research High Performance Computing Specialist > SDSU UNSS > SAD 119, Box 2201, Brookings, SD 57007-1598 > 605.688.4679 (wk) / 605.521.3062 (cell) > > > moore at moore-pc2:~/petsc-3.3-p4/src/ksp/ksp/examples/tutorials> make runex11 > 1,8c1,20 > < 0 KSP Residual norm 4.62271 > < 1 KSP Residual norm 1.58711 > < 2 KSP Residual norm 0.767563 > < 3 KSP Residual norm 0.472102 > < 4 KSP Residual norm 0.435655 > < 5 KSP Residual norm 0.154866 > < 6 KSP Residual norm < 1.e-11 > < Norm of error < 1.e-12 iterations 6 > --- > > 1+1i > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > > [0]PETSC ERROR: Object is in wrong state! > > [0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatGetOwnershipRange()! > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 4, Fri Oct 26 10:46:51 CDT 2012 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./ex11 on a arch-linu named moore-pc2 by moore Mon Oct 29 15:27:59 2012 > > [0]PETSC ERROR: Libraries linked from /home/moore/petsc-3.3-p4/arch-linux2-c-debug/lib > > [0]PETSC ERROR: Configure run at Mon Oct 29 15:00:29 2012 > > [0]PETSC ERROR: Configure options --download-f-blas-lapack=1 --with-shared-libraries=1 --download-mpich --with-scalar-type=complex > > [0]PETSC ERROR: ------------------------------------------------------------------------ > > [0]PETSC ERROR: MatGetOwnershipRange() line 5992 in /home/moore/petsc-3.3-p4/src/mat/interface/matrix.c > > [0]PETSC ERROR: main() line 83 in src/ksp/ksp/examples/tutorials/ex11.c > > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > > [cli_0]: aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 > /home/moore/petsc-3.3-p4/src/ksp/ksp/examples/tutorials > Possible problem with with ex11_1, diffs above \n========================================= > > > > > -------------- next part -------------- diff --git a/src/ksp/ksp/examples/tutorials/ex11.c b/src/ksp/ksp/examples/tutorials/ex11.c --- a/src/ksp/ksp/examples/tutorials/ex11.c +++ b/src/ksp/ksp/examples/tutorials/ex11.c @@ -74,6 +74,7 @@ ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,dim,dim);CHKERRQ(ierr); ierr = MatSetFromOptions(A);CHKERRQ(ierr); + ierr = MatSetUp(A);CHKERRQ(ierr); /* Currently, all PETSc parallel matrix formats are partitioned by From knepley at gmail.com Mon Oct 29 19:48:34 2012 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Oct 2012 20:48:34 -0400 Subject: [petsc-users] AOCreateBasic In-Reply-To: References: Message-ID: On Mon, Oct 29, 2012 at 3:51 PM, Eduardo wrote: > Hi, > Can I use AOCreateBasic to create a mapping that has negative indexes? > I mean, a mapping that starts at some -N and goes up to napp-1-N ? > With no holes. > No, it relies on being a permutation. It would be easy to just shift the order that comes out however. Matt > Thanks in advance, > Eduardo > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 30 06:41:33 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 30 Oct 2012 12:41:33 +0100 Subject: [petsc-users] How rto use multigrid? In-Reply-To: References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> Message-ID: <508FBCED.3010701@gmail.com> On 28/10/2012 2:17 PM, Jed Brown wrote: > > Algebraic multigrid can be used directly, -pc_type gamg > -pc_gamg_agg_nsmooths 1. Geometric either required that you use the > PCMG interface to set interpolation (and provide a coarse operator for > non-Galerkin) or use a DM that provides coarsening capability. > > What kind of problem are you solving? > Hi, May I know if there is an example which explains using the geometric MG, either using PCMG or DM? Yours sincerely, TAY wee-beng > On Oct 28, 2012 6:09 AM, "w_ang_temp" > wrote: > > Hello, > I want to use the multigrid as a preconditioner. The > introduction about it in the manual is little. > So are there some typical examples or details about multigrid? Is > it used just like other preconditioners > like jacobi, sor, which can be simply used in the cammand line > options? > Thanks. > Jim > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 30 07:44:49 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Oct 2012 08:44:49 -0400 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <508FBCED.3010701@gmail.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <508FBCED.3010701@gmail.com> Message-ID: On Tue, Oct 30, 2012 at 7:41 AM, TAY wee-beng wrote: > On 28/10/2012 2:17 PM, Jed Brown wrote: > > Algebraic multigrid can be used directly, -pc_type gamg > -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG > interface to set interpolation (and provide a coarse operator for > non-Galerkin) or use a DM that provides coarsening capability. > > What kind of problem are you solving? > > > Hi, > > May I know if there is an example which explains using the geometric MG, > either using PCMG or DM? > SNES ex5 and ex19 both use MG. Matt > Yours sincerely, > > TAY wee-beng > > > On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: > >> Hello, >> I want to use the multigrid as a preconditioner. The introduction >> about it in the manual is little. >> So are there some typical examples or details about multigrid? Is it used >> just like other preconditioners >> like jacobi, sor, which can be simply used in the cammand line options? >> Thanks. >> Jim >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Tue Oct 30 10:27:04 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Tue, 30 Oct 2012 23:27:04 +0800 (CST) Subject: [petsc-users] When is it better to use MUMPS? Message-ID: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> Hello, As is know, MUMPS is based on a direct method. When the system is big, the parallel direct solver is not effective compared with the iterative solver. So when is it better to use MUMPS? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Tue Oct 30 10:22:51 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Tue, 30 Oct 2012 23:22:51 +0800 (CST) Subject: [petsc-users] When is it better to use MUMPS? Message-ID: <3b939b74.28eb1.13ab2449aa9.Coremail.w_ang_temp@163.com> Hello, As is know, MUMPS is based on a direct method. When the system is big, the parallel direct solver is not effective compared with the iterative solver. So when is it better to use MUMPS? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From jedbrown at mcs.anl.gov Tue Oct 30 10:47:59 2012 From: jedbrown at mcs.anl.gov (Jed Brown) Date: Tue, 30 Oct 2012 08:47:59 -0700 Subject: [petsc-users] When is it better to use MUMPS? In-Reply-To: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> References: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> Message-ID: Simple: it's better when it's faster. Small problems and moderate-size 2D problems often work well with direct solvers. On Oct 30, 2012 8:29 AM, "w_ang_temp" wrote: > Hello, > As is know, MUMPS is based on a direct method. When the system is big, > the parallel direct solver is > not effective compared with the iterative solver. So when is it better to > use MUMPS? > Thanks. > Jim > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zonexo at gmail.com Tue Oct 30 17:34:28 2012 From: zonexo at gmail.com (TAY wee-beng) Date: Tue, 30 Oct 2012 23:34:28 +0100 Subject: [petsc-users] How rto use multigrid? In-Reply-To: References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <508FBCED.3010701@gmail.com> Message-ID: <509055F4.1050806@gmail.com> On 30/10/2012 1:44 PM, Matthew Knepley wrote: > On Tue, Oct 30, 2012 at 7:41 AM, TAY wee-beng > wrote: > > On 28/10/2012 2:17 PM, Jed Brown wrote: >> >> Algebraic multigrid can be used directly, -pc_type gamg >> -pc_gamg_agg_nsmooths 1. Geometric either required that you use >> the PCMG interface to set interpolation (and provide a coarse >> operator for non-Galerkin) or use a DM that provides coarsening >> capability. >> >> What kind of problem are you solving? >> > > Hi, > > May I know if there is an example which explains using the > geometric MG, either using PCMG or DM? > > > SNES ex5 and ex19 both use MG. > > Matt Are there examples which uses ksp and geometric MG? I want to solve a Poisson eqn which has been discretized to give a sys of linear eqns. I also found this thread : http://lists.mcs.anl.gov/pipermail/petsc-users/2012-August/015073.html Is it using geometric MG to solve a sys of linear eqns? > > Yours sincerely, > > TAY wee-beng > > >> On Oct 28, 2012 6:09 AM, "w_ang_temp" > > wrote: >> >> Hello, >> I want to use the multigrid as a preconditioner. The >> introduction about it in the manual is little. >> So are there some typical examples or details about >> multigrid? Is it used just like other preconditioners >> like jacobi, sor, which can be simply used in the cammand >> line options? >> Thanks. >> Jim >> >> > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhaonanavril at gmail.com Tue Oct 30 17:36:32 2012 From: zhaonanavril at gmail.com (NAN ZHAO) Date: Tue, 30 Oct 2012 17:36:32 -0500 Subject: [petsc-users] matzeroentries usage Message-ID: Dear all, I need to solve Ax=b in a iteratively, and the A is b is ajusted in each iteration. I am using matzeroentries to zeroout the values in A (A is in MPIBAIJ format). Somehow, I found this cause me some unexpected wrong solutions. I do not want to retain the non-zero structures in A when I try to zero it. Does anyone have a idea to do that. Or I need to destroy the A matrix each time I give new values to it (the non-zero structure is changing). Thanks, Nan -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 30 17:44:09 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Oct 2012 18:44:09 -0400 Subject: [petsc-users] matzeroentries usage In-Reply-To: References: Message-ID: On Tue, Oct 30, 2012 at 6:36 PM, NAN ZHAO wrote: > Dear all, > > I need to solve Ax=b in a iteratively, and the A is b is ajusted in each > iteration. I am using matzeroentries to zeroout the values in A (A is in > MPIBAIJ format). Somehow, I found this cause me some unexpected wrong > solutions. I do not want to retain the non-zero structures in A when I try > to zero it. Does anyone have a idea to do that. Or I need to destroy the A > matrix each time I give new values to it (the non-zero structure is > changing). > If the nonzero structure of A is changing, there is really no advantage to keeping it (unless you can see one). I would recreate it and call KSPSetOperators() again. Matt > > Thanks, > Nan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Oct 30 17:49:33 2012 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 30 Oct 2012 18:49:33 -0400 Subject: [petsc-users] How rto use multigrid? In-Reply-To: <509055F4.1050806@gmail.com> References: <2dce573b.e609.13aa77d4c8e.Coremail.w_ang_temp@163.com> <508FBCED.3010701@gmail.com> <509055F4.1050806@gmail.com> Message-ID: On Tue, Oct 30, 2012 at 6:34 PM, TAY wee-beng wrote: > On 30/10/2012 1:44 PM, Matthew Knepley wrote: > > On Tue, Oct 30, 2012 at 7:41 AM, TAY wee-beng wrote: > >> On 28/10/2012 2:17 PM, Jed Brown wrote: >> >> Algebraic multigrid can be used directly, -pc_type gamg >> -pc_gamg_agg_nsmooths 1. Geometric either required that you use the PCMG >> interface to set interpolation (and provide a coarse operator for >> non-Galerkin) or use a DM that provides coarsening capability. >> >> What kind of problem are you solving? >> >> >> Hi, >> >> May I know if there is an example which explains using the geometric MG, >> either using PCMG or DM? >> > > SNES ex5 and ex19 both use MG. > > Matt > > Are there examples which uses ksp and geometric MG? I want to solve a > Poisson eqn which has been discretized to give a sys of linear eqns. > This is geometric MG. Did you run any of the tutorial runs? For example, src/snes/examples/tutorials/makefile has runex5: -@${MPIEXEC} -n 1 ./ex5 -pc_type mg -ksp_monitor_short -snes_view -pc_mg_levels 3 -pc_mg_galerkin -da_grid_x 17 -da_grid_y 17 -mg_levels_ksp_monitor_short -mg_levels_ksp_norm_type unpreconditioned -snes_monitor_short -mg_levels_ksp_chebyshev_estimate_eigenvalues 0.5,1.1 -mg_levels_pc_type sor -pc_mg_type full > ex5_1.tmp 2>&1; \ if (${DIFF} output/ex5_1.out ex5_1.tmp) then true; \ else echo ${PWD} "\nPossible problem with with ex5, diffs above \n========================================="; fi; \ ${RM} -f ex5_1.tmp and many more. Matt > I also found this thread : > > http://lists.mcs.anl.gov/pipermail/petsc-users/2012-August/015073.html > > Is it using geometric MG to solve a sys of linear eqns? > > > >> Yours sincerely, >> >> TAY wee-beng >> >> >> On Oct 28, 2012 6:09 AM, "w_ang_temp" wrote: >> >>> Hello, >>> I want to use the multigrid as a preconditioner. The introduction >>> about it in the manual is little. >>> So are there some typical examples or details about multigrid? Is it >>> used just like other preconditioners >>> like jacobi, sor, which can be simply used in the cammand line options? >>> Thanks. >>> >>> Jim >>> >>> >>> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Tue Oct 30 17:50:59 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Tue, 30 Oct 2012 17:50:59 -0500 Subject: [petsc-users] matzeroentries usage In-Reply-To: References: Message-ID: NAN : After reset matrix A, you must call KSPSetOperators(ksp,...) again. See petsc-dev/src/ksp/ksp/examples/tutorials/ex5.c on how to solve modified matrices. Hong > > I need to solve Ax=b in a iteratively, and the A is b is ajusted in each > iteration. I am using matzeroentries to zeroout the values in A (A is in > MPIBAIJ format). Somehow, I found this cause me some unexpected wrong > solutions. I do not want to retain the non-zero structures in A when I try > to zero it. Does anyone have a idea to do that. Or I need to destroy the A > matrix each time I give new values to it (the non-zero structure is > changing). > > Thanks, > Nan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Wed Oct 31 08:02:12 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Wed, 31 Oct 2012 21:02:12 +0800 (CST) Subject: [petsc-users] When is it better to use MUMPS? In-Reply-To: References: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> Message-ID: <2c627979.a56d.13ab6ea309e.Coremail.w_ang_temp@163.com> Hello, Jed In my view, the intent of parallel computing is sovling large system. As you said, MUMPS is mainly used in small problems. So are they contradictory. Or, is the "Small problems and moderate-size 2D problems" just relative, compared with the large system? Thanks. Jim At 2012-10-30 23:47:59,"Jed Brown" wrote: Simple: it's better when it's faster. Small problems and moderate-size 2D problems often work well with direct solvers. On Oct 30, 2012 8:29 AM, "w_ang_temp" wrote: Hello, As is know, MUMPS is based on a direct method. When the system is big, the parallel direct solver is not effective compared with the iterative solver. So when is it better to use MUMPS? Thanks. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Wed Oct 31 08:24:03 2012 From: hzhang at mcs.anl.gov (Hong Zhang) Date: Wed, 31 Oct 2012 08:24:03 -0500 Subject: [petsc-users] When is it better to use MUMPS? In-Reply-To: <2c627979.a56d.13ab6ea309e.Coremail.w_ang_temp@163.com> References: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> <2c627979.a56d.13ab6ea309e.Coremail.w_ang_temp@163.com> Message-ID: Jim, In my view, the intent of parallel computing is sovling large system. > As you said, MUMPS is mainly used in > " the intent" does not automatically become the reality. All parallel direct solvers are intended to solve ill-conditioned problems as large as possible. But the algorithms (full matrix factorization) consume large memories and inter-processor communications. Unless your matrix has special data structure, the direct solvers cannot be scalable in general. small problems. So are they contradictory. Or, is the "Small problems and > moderate-size 2D problems" just relative, > compared with the large system? > Yes. Hong > > > > > > > At 2012-10-30 23:47:59,"Jed Brown" wrote: > > Simple: it's better when it's faster. Small problems and moderate-size 2D > problems often work well with direct solvers. > On Oct 30, 2012 8:29 AM, "w_ang_temp" wrote: > >> Hello, >> As is know, MUMPS is based on a direct method. When the system is >> big, the parallel direct solver is >> not effective compared with the iterative solver. So when is it better to >> use MUMPS? >> Thanks. >> Jim >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From w_ang_temp at 163.com Wed Oct 31 08:49:30 2012 From: w_ang_temp at 163.com (w_ang_temp) Date: Wed, 31 Oct 2012 21:49:30 +0800 (CST) Subject: [petsc-users] When is it better to use MUMPS? In-Reply-To: References: <79882a62.21ea0.13ab24877b3.Coremail.w_ang_temp@163.com> <2c627979.a56d.13ab6ea309e.Coremail.w_ang_temp@163.com> Message-ID: <3129574c.10528.13ab7157fc7.Coremail.w_ang_temp@163.com> Thanks. I get it. >? 2012-10-31 21:24:03?"Hong Zhang" ??? >Jim, >> In my view, the intent of parallel computing is sovling large system. As you said, MUMPS is mainly used in >" the intent" does not automatically become the reality. >All parallel direct solvers are intended to solve ill-conditioned problems as large as possible. >But the algorithms (full matrix factorization) consume large memories and inter-processor >communications. Unless your matrix has special data structure, the direct solvers cannot be scalable >in general. >>small problems. So are they contradictory. Or, is the "Small problems and moderate-size 2D problems" just relative, >>compared with the large system? >Yes. >Hong >>At 2012-10-30 23:47:59,"Jed Brown" wrote: >>Simple: it's better when it's faster. Small problems and moderate-size 2D problems often work well with direct solvers. >>On Oct 30, 2012 8:29 AM, "w_ang_temp" wrote: >>Hello, >>As is know, MUMPS is based on a direct method. When the system is big, the parallel direct solver is >>not effective compared with the iterative solver. So when is it better to use MUMPS? >> Thanks. >> Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: