[petsc-users] Fwd: Re: A question on the PETSc options for non-uniform grid
Zhenglun (Alan) Wei
zhenglun.wei at gmail.com
Fri Apr 5 11:59:32 CDT 2013
Dear folks,
I forget to forward my reply to the list. Please check the
'Forwarded Message' about my code of Poisson solver with non-uniform
grid and the procedure to reproduce the error for 'gamg'.
However, here I have some questions on the HYPRE preconditioner for
the GMRES. One of my friend suggested me to add this to my code:
ierr = KSPGetPC(solver, &pc);CHKERRQ(ierr);
ierr = PCSetType(pc, PCHYPRE);CHKERRQ(ierr);
ierr = PCHYPRESetType(pc,"boomeramg");CHKERRQ(ierr);
ierr = PetscOptionsSetValue("-pc_hypre_boomeramg_max_levels", "25");
CHKERRQ(ierr);
ierr = PetscOptionsSetValue("-pc_hypre_boomeramg_strong_threshold",
"0.0"); CHKERRQ(ierr);
ierr = PetscOptionsSetValue("-pc_hypre_boomeramg_relax_type_all",
"SOR/Jacobi"); CHKERRQ(ierr);
After this is done, the time for solving the Poisson equation indeed
reduced to half; however, it is still very slow. Here I attached output
file (out_GMRES_HYPRE-pc in the attachment) with -ksp_monitor and
--ksp_view. Could you please help me to take a look and see which part I
can modified to speed up the solver.
thanks,
Alan
-------- Forwarded Message --------
主题: Re: [petsc-users] A question on the PETSc options for non-uniform
grid
日期: Sun, 31 Mar 2013 18:21:40 -0500
发件人: Zhenglun (Alan) Wei <zhenglun.wei at gmail.com>
收件人: Jed Brown <jedbrown at mcs.anl.gov>, "Mark F. Adams"
<mark.adams at columbia.edu>
Dear Dr. Brown,
Thank you so much for your answers. Here are my reply.
I reduced my code and attached here. It derives from the KSP ex45.c plus
my own code for the non uniform grid. The boundary condtion follows the
original method of ex45.c. Firstly, I should mention that if the code
runs for uniform grid, it is very fine. In the 'ITTCRun' script, there
are four executable lines.
1, it uses ksp = 'CG' and pc = 'GAMG'. It will comes up the
'un-symmetric graphic' problem;
2, it adds '-pc_gamg_sym_graph true' at the end. As I mentioned before,
it provides a very crazy norm since the '-pc_gamg_sym_graph true' is
used. The 'out_ksp=CG' shows the log and the norm.
3, No matter if the ksp type changed to 'GMRES' (which is the 3rd
executable line) or '-pc_gamg_sym_graph true' changed to
'-pc_gamg_threhold 0.0', the code will stacks there. However, it seems
not to be a 'deadlock' because if I reduce the computational load, i.e.
mesh size, it will complete but just very slow.
4, it uses everything for the default; I believe it is ksp = 'GMRES'
without any preconditioner. It can solve the fine mesh problem yet just
very slow.
I really appreciate your time and help, :)
Alan
> "Zhenglun (Alan) Wei" <zhenglun.wei at gmail.com> writes:
>
>> Dear All,
>> I hope you're having a nice day.
>> Based on ksp ex45, a 3D Poisson solver with non-uniform grid is coded.
>> The PETSc options I used is:
>> /mpiexec -np 32 ./ex45 -pc_type gamg -ksp_type cg -pc_gamg_type agg
>> -pc_gamg_agg_nsmooths 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type
>> richardson -ksp_rtol 1.0e-7/
>> There are some problems:
>> 1, if the mesh is very coarse with very small amount of the grid, the
>> code runs well;
>> 2, if the mesh is fine, an error message comes up saying 'un-symmetric
>> graph'. It suggests me to use '-pc_gamg_sym_graph true' or
>> '-pc_gamg_thredhold 0.0'.
> How are you implementing boundary conditions?
>
>> a) if '-pc_gamg_sym_graph true' is used, the code runs but blows up very
>> quickly with crazy norm;
>> b) if '-pc_gamg_thredhold 0.0' is used, the code stops on the KSPSolve()
>> forever;
>> 3, because of the 'un-symmetric graph' error, it reminds me that the
>> matrix may not be symmetric, which indicates that '-ksp_type cg' may not
>> be a good option. Therefore, I changed it to '-ksp_type gmres'. It makes
>> the code run with 'moderate fine' mesh. However, it stops at KSPSolve()
>> also with fine mesh.
>> a) does there any other '-ksp_type' fit better for this case?
>> b) does multigrid preconditioner work well for gmres?
> All of the above should work. Can you reproduce with our version of
> ex45.c, or can you send your version with the full error messages you
> are seeing (or other symptoms, like deadlock?) and instructions to
> reproduce?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130405/058109b6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NonUniformPoisson.zip
Type: application/zip
Size: 10573 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130405/058109b6/attachment-0001.zip>
-------------- next part --------------
NumRank = 32
Ready for BuildingNonUniformGrid!!
Ready for ApplyNonUniformGrid#1!!
Ready for Initialization!!
Ready for the main iteration!!
Ready for ApplyNonUniform of the KSPSolve#1!!
Ready for ComputePpRHS of the KSPSolve#1!!
Ready for ComputeMatrix of the KSPSolve#1!!
Ready for KspSetDM of the KSPSolve#1!!
Ready for KSPSetFromOptions of the KSPSolve#1!!
Ready for KSPSetUp of the KSPSolve#1!!
Finish KSPSetUp of the KSPSolve#1!!
Ready for KSPSolve#1!!
Finish KSPSolve#1!!
Residual norm 121.02
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
./ex45 on a linux-gnu-c-nodebug named n028 with 32 processors, by zlwei Sun Mar 31 18:01:09 2013
Using Petsc Development HG revision: 60bf177dda1f5f14babbace55c4a430ac46ab2a6 HG Date: Thu Feb 14 11:32:41 2013 -0600
Max Max/Min Avg Total
Time (sec): 2.793e+01 1.00005 2.793e+01
Objects: 5.510e+02 1.00364 5.491e+02
Flops: 5.698e+08 1.22785 5.141e+08 1.645e+10
Flops/sec: 2.040e+07 1.22784 1.841e+07 5.890e+08
MPI Messages: 4.776e+03 2.14822 3.434e+03 1.099e+05
MPI Message Lengths: 4.300e+07 1.63512 1.016e+04 1.117e+09
MPI Reductions: 1.139e+03 1.00176
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 1.1397e-01 0.4% 2.0336e+08 1.2% 1.280e+02 0.1% 5.241e+01 0.5% 3.000e+00 0.3%
1: DMMG Setup: 4.1170e-01 1.5% 0.0000e+00 0.0% 1.360e+03 1.2% 1.597e+02 1.6% 3.700e+01 3.2%
2: Pressure RHS Setup: 2.6268e+01 94.0% 1.3900e+10 84.5% 9.833e+04 89.5% 9.435e+03 92.9% 1.074e+03 94.3%
3: Pressure Solve: 1.1388e+00 4.1% 2.3473e+09 14.3% 1.008e+04 9.2% 5.131e+02 5.1% 2.206e+01 1.9%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flops: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %f - percent flops in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flops --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %f %M %L %R %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecNorm 1 1.0 2.2058e-0218.1 7.50e+05 1.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 7 12 0 0 33 1088
VecAXPY 1 1.0 1.0166e-02 5.0 7.50e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 6 12 0 0 0 2361
VecScatterBegin 1 1.0 2.9681e-03 6.1 0.00e+00 0.0 1.3e+02 4.5e+04 0.0e+00 0 0 0 1 0 2 0100100 0 0
VecScatterEnd 1 1.0 1.7600e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0
MatMult 1 1.0 6.9038e-02 1.4 4.87e+06 1.0 1.3e+02 4.5e+04 0.0e+00 0 1 0 1 0 54 76100100 0 2250
--- Event Stage 1: DMMG Setup
VecSet 1 1.0 2.9253e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 6 0 0 0 0 0
VecScatterBegin 1 1.0 3.5793e-02 1.1 0.00e+00 0.0 3.7e+02 3.2e+04 0.0e+00 0 0 0 1 0 8 0 27 67 0 0
VecScatterEnd 1 1.0 1.8298e-02 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 0 0 0 0 0
--- Event Stage 2: Pressure RHS Setup
KSPGMRESOrthog 40 1.0 8.6936e-01 1.4 9.39e+07 1.0 0.0e+00 0.0e+00 4.0e+01 3 18 0 0 4 3 21 0 0 4 3402
KSPSetUp 10 1.0 9.0960e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 2 0 0 0 0 2 0
VecMDot 40 1.0 6.0814e-01 1.7 4.69e+07 1.0 0.0e+00 0.0e+00 4.0e+01 2 9 0 0 4 2 11 0 0 4 2431
VecNorm 44 1.0 2.4593e-01 2.2 9.39e+06 1.0 0.0e+00 0.0e+00 4.4e+01 1 2 0 0 4 1 2 0 0 4 1202
VecScale 44 1.0 7.4415e-02 4.9 4.69e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1987
VecCopy 4 1.0 9.3884e-03 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 74 1.0 6.2634e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 4 1.0 1.1530e-02 3.2 8.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2332
VecMAXPY 44 1.0 3.7019e-01 1.2 5.55e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 11 0 0 0 1 13 0 0 0 4720
VecAssemblyBegin 64 1.0 3.5270e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+02 1 0 0 0 17 1 0 0 0 18 0
VecAssemblyEnd 64 1.0 3.2663e-04 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 44 1.0 1.5967e-01 1.2 4.69e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 926
VecScatterBegin 108 1.0 1.2571e-01 1.7 0.00e+00 0.0 4.5e+04 7.9e+03 0.0e+00 0 0 41 32 0 0 0 46 34 0 0
VecScatterEnd 108 1.0 4.2095e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
VecSetRandom 4 1.0 1.9181e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 44 1.0 3.0430e-01 1.9 1.41e+07 1.0 0.0e+00 0.0e+00 4.4e+01 1 3 0 0 4 1 3 0 0 4 1458
MatMult 40 1.0 9.8190e-01 1.2 9.89e+07 1.3 1.5e+04 6.5e+03 0.0e+00 3 17 13 8 0 4 20 15 9 0 2863
MatConvert 4 1.0 2.4130e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 1 0 0 0 0 1 0 0 0 0 0
MatScale 12 1.0 2.4393e-01 1.2 1.46e+07 1.3 1.5e+03 6.5e+03 0.0e+00 1 3 1 1 0 1 3 1 1 0 1701
MatAssemblyBegin 66 1.0 1.0796e+00 1.6 0.00e+00 0.0 1.0e+04 1.3e+04 7.2e+01 3 0 9 12 6 3 0 10 13 7 0
MatAssemblyEnd 66 1.0 1.8091e+00 1.1 0.00e+00 0.0 1.7e+04 1.4e+03 2.2e+02 6 0 16 2 19 7 0 18 2 20 0
MatGetRow 1707200 1.0 3.9819e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 15 0 0 0 0 0
MatCoarsen 4 1.0 7.2832e-01 1.0 0.00e+00 0.0 1.7e+04 7.1e+03 1.4e+02 3 0 15 11 12 3 0 17 12 13 0
MatAXPY 4 1.0 5.4486e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatTranspose 4 1.0 1.1323e+00 1.0 0.00e+00 0.0 1.1e+04 9.8e+03 6.4e+01 4 0 10 10 6 4 0 11 10 6 0
MatMatMult 4 1.0 8.7380e-01 1.1 1.03e+07 1.3 8.7e+03 3.3e+03 9.6e+01 3 2 8 3 8 3 2 9 3 9 337
MatPtAP 4 1.0 3.0574e+00 1.0 1.82e+08 2.1 1.4e+04 1.3e+04 1.0e+02 11 26 13 16 9 12 30 14 18 9 1382
MatTrnMatMult 4 1.0 5.4484e+00 1.0 9.66e+07 1.8 8.3e+03 3.7e+04 1.2e+02 19 14 8 27 10 21 17 8 29 11 424
MatGetLocalMat 20 1.0 4.8721e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 2 0 0 0 2 2 0 0 0 2 0
MatGetBrAoCol 12 1.0 2.6553e-01 1.6 0.00e+00 0.0 1.0e+04 1.2e+04 1.6e+01 1 0 9 11 1 1 0 10 12 1 0
MatGetSymTrans 8 1.0 4.9516e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCSetUp 1 1.0 2.4610e+01 1.0 4.86e+08 1.2 9.7e+04 1.1e+04 1.0e+03 88 84 88 91 88 94100 98 98 94 565
PCGAMGgraph_AGG 4 1.0 8.8514e+00 1.0 1.03e+07 1.3 1.8e+04 8.8e+03 1.4e+02 32 2 17 14 12 34 2 19 15 13 33
PCGAMGcoarse_AGG 4 1.0 6.6358e+00 1.0 9.66e+07 1.8 3.4e+04 1.5e+04 3.4e+02 24 14 31 45 29 25 17 34 49 31 348
PCGAMGProl_AGG 4 1.0 5.1585e-01 1.1 0.00e+00 0.0 7.6e+03 5.5e+03 1.1e+02 2 0 7 4 10 2 0 8 4 10 0
PCGAMGPOpt_AGG 4 1.0 5.5509e+00 1.0 2.36e+08 1.1 2.3e+04 5.3e+03 2.2e+02 20 43 21 11 20 21 51 24 12 21 1274
--- Event Stage 3: Pressure Solve
KSPSetUp 2 1.0 7.1526e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 1 1.0 1.1429e+00 1.0 8.00e+07 1.2 1.0e+04 5.6e+03 2.2e+01 4 14 9 5 2 100100100100100 2054
VecTDot 4 1.0 6.5522e-02 2.4 3.00e+06 1.0 0.0e+00 0.0e+00 4.0e+00 0 1 0 0 0 4 4 0 0 18 1465
VecNorm 2 1.0 4.0948e-02 8.3 1.50e+06 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 2 2 0 0 9 1172
VecCopy 10 1.0 3.5149e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0
VecSet 34 1.0 2.3161e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0
VecAXPY 18 1.0 5.9955e-02 1.5 4.91e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 4 7 0 0 0 2594
VecAYPX 17 1.0 4.8838e-02 1.2 2.46e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 3 0 0 0 1592
VecAssemblyBegin 1 1.0 3.1125e-02 7.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 1 0 0 0 14 0
VecAssemblyEnd 1 1.0 1.5974e-05 8.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 16 1.0 5.5181e-02 1.2 1.71e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 5 2 0 0 0 974
VecScatterBegin 34 1.0 2.6413e-02 2.1 0.00e+00 0.0 1.0e+04 5.6e+03 0.0e+00 0 0 9 5 0 2 0100100 0 0
VecScatterEnd 34 1.0 7.9229e-02 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 4 0 0 0 0 0
MatMult 18 1.0 5.3817e-01 1.1 4.93e+07 1.2 6.1e+03 8.1e+03 0.0e+00 2 9 6 4 0 45 61 60 87 0 2667
MatMultAdd 8 1.0 1.0760e-01 1.2 8.54e+06 1.3 2.0e+03 1.8e+03 0.0e+00 0 1 2 0 0 9 10 20 6 0 2236
MatMultTranspose 8 1.0 1.1546e-01 1.4 8.54e+06 1.3 2.0e+03 1.8e+03 0.0e+00 0 1 2 0 0 8 10 20 6 0 2084
MatSolve 2 0.0 2.0981e-05 0.0 5.70e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 272
MatLUFactorSym 1 1.0 4.7207e-05 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 14 0
MatLUFactorNum 1 1.0 7.2956e-0525.5 3.59e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 492
MatGetRowIJ 1 0.0 8.8215e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetOrdering 1 0.0 7.1049e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e-01 0 0 0 0 0 0 0 0 0 1 0
PCSetUp 1 1.0 6.8903e-04 7.5 3.59e+04 0.0 0.0e+00 0.0e+00 5.1e+00 0 0 0 0 0 0 0 0 0 23 52
PCSetUpOnBlocks 2 1.0 7.0190e-04 7.1 3.59e+04 0.0 0.0e+00 0.0e+00 5.1e+00 0 0 0 0 0 0 0 0 0 23 51
PCApply 2 1.0 8.5915e-01 1.1 6.35e+07 1.3 9.8e+03 4.6e+03 1.3e+01 3 11 9 4 1 73 78 97 80 59 2119
--- Event Stage 4: Unknown
--- Event Stage 5: Unknown
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 0 8 26784 0
DMKSP interface 0 1 656 0
Vector 1 46 72328368 0
Vector Scatter 0 17 18020 0
Matrix 0 48 132500240 0
Distributed Mesh 0 4 18810208 0
Bipartite Graph 0 8 6400 0
Index Set 0 5 4144 0
IS L to G Mapping 0 6 12530760 0
Preconditioner 0 7 7052 0
Viewer 1 0 0 0
--- Event Stage 1: DMMG Setup
Krylov Solver 1 0 0 0
Vector 6 4 6080 0
Vector Scatter 4 0 0 0
Distributed Mesh 2 0 0 0
Bipartite Graph 4 0 0 0
Index Set 10 10 3138568 0
IS L to G Mapping 3 0 0 0
--- Event Stage 2: Pressure RHS Setup
Krylov Solver 12 5 121792 0
DMKSP interface 1 0 0 0
Vector 223 184 88029856 0
Vector Scatter 36 23 24380 0
Matrix 123 76 527242404 0
Matrix Coarsen 4 4 2544 0
Distributed Mesh 3 1 4368 0
Bipartite Graph 10 6 5024 0
Index Set 78 76 3772892 0
IS L to G Mapping 3 0 0 0
Preconditioner 12 5 4520 0
PetscRandom 4 4 2528 0
--- Event Stage 3: Pressure Solve
Vector 4 0 0 0
Matrix 1 0 0 0
Index Set 5 2 1600 0
--- Event Stage 4: Unknown
--- Event Stage 5: Unknown
========================================================================================================================
Average time to get PetscTime(): 5.00679e-07
Average time for MPI_Barrier(): 0.000554371
Average time for zero size MPI_Send(): 0.000241652
#PETSc Option Table entries:
-ksp_rtol 1.0e-7
-ksp_type cg
-log_summary
-mg_levels_ksp_max_it 1
-mg_levels_ksp_type richardson
-pc_gamg_agg_nsmooths 1
-pc_gamg_sym_graph true
-pc_gamg_type agg
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Thu Feb 14 14:17:01 2013
Configure options: --download-f-blas-lapack --download-mpich --with-debugging=0 PETSC_ARCH=linux-gnu-c-nodebug
-----------------------------------------
Libraries compiled on Thu Feb 14 14:17:01 2013 on login1.ittc.ku.edu
Machine characteristics: Linux-2.6.32-220.13.1.el6.x86_64-x86_64-with-redhat-6.2-Santiago
Using PETSc directory: /bio/work1/zlwei/PETSc/petsc-dev
Using PETSc arch: linux-gnu-c-nodebug
-----------------------------------------
Using C compiler: /bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/bin/mpif90 -fPIC -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------
Using include paths: -I/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/include -I/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/include
-----------------------------------------
Using C linker: /bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/bin/mpicc
Using Fortran linker: /bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/bin/mpif90
Using libraries: -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/lib -lpetsc -Wl,-rpath,/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/lib -L/bio/work1/zlwei/PETSc/petsc-dev/linux-gnu-c-nodebug/lib -lflapack -lfblas -lX11 -lpthread -lm -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -lmpichf90 -lgfortran -lm -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl
-----------------------------------------
-------------- next part --------------
NumRank = 32
Ready for BuildingNonUniformGrid!!
Ready for ApplyNonUniformGrid#1!!
rank = 25, NbrRanks are: L: 24, R: 26, B:21, T:29, A:17, F:33!!
rank = 25: xs = 75, ys = 0, zs = 150, mx = 300, my = 200, mz = 200!
rank = 25: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1
rank = 25: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 31, NbrRanks are: L: 30, R: 32, B:27, T:35, A:23, F:39!!
rank = 31: xs = 225, ys = 100, zs = 150, mx = 300, my = 200, mz = 200!
rank = 31: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1
rank = 31: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 28, NbrRanks are: L: 27, R: 29, B:24, T:32, A:20, F:36!!
rank = 28: xs = 0, ys = 100, zs = 150, mx = 300, my = 200, mz = 200!
rank = 28: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1
rank = 28: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 29, NbrRanks are: L: 28, R: 30, B:25, T:33, A:21, F:37!!
rank = 29: xs = 75, ys = 100, zs = 150, mx = 300, my = 200, mz = 200!
rank = 29: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1
rank = 29: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 24, NbrRanks are: L: 23, R: 25, B:20, T:28, A:16, F:32!!
rank = 24: xs = 0, ys = 0, zs = 150, mx = 300, my = 200, mz = 200!
rank = 24: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1
rank = 24: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 26, NbrRanks are: L: 25, R: 27, B:22, T:30, A:18, F:34!!
rank = 26: xs = 150, ys = 0, zs = 150, mx = 300, my = 200, mz = 200!
rank = 26: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1
rank = 26: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 27, NbrRanks are: L: 26, R: 28, B:23, T:31, A:19, F:35!!
rank = 27: xs = 225, ys = 0, zs = 150, mx = 300, my = 200, mz = 200!
rank = 27: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 1
rank = 27: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 20, NbrRanks are: L: 19, R: 21, B:16, T:24, A:12, F:28!!
rank = 20: xs = 0, ys = 100, zs = 100, mx = 300, my = 200, mz = 200!
rank = 20: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 20: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 30, NbrRanks are: L: 29, R: 31, B:26, T:34, A:22, F:38!!
rank = 30: xs = 150, ys = 100, zs = 150, mx = 300, my = 200, mz = 200!
rank = 30: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 1
rank = 30: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 4, NbrRanks are: L: 3, R: 5, B:0, T:8, A:-4, F:12!!
rank = 4: xs = 0, ys = 100, zs = 0, mx = 300, my = 200, mz = 200!
rank = 4: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0
rank = 4: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 12, NbrRanks are: L: 11, R: 13, B:8, T:16, A:4, F:20!!
rank = 12: xs = 0, ys = 100, zs = 50, mx = 300, my = 200, mz = 200!
rank = 12: left BC = 1, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 12: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 19, NbrRanks are: L: 18, R: 20, B:15, T:23, A:11, F:27!!
rank = 19: xs = 225, ys = 0, zs = 100, mx = 300, my = 200, mz = 200!
rank = 19: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 19: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 1, NbrRanks are: L: 0, R: 2, B:-3, T:5, A:-7, F:9!!
rank = 1: xs = 75, ys = 0, zs = 0, mx = 300, my = 200, mz = 200!
rank = 1: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0
rank = 18, NbrRanks are: L: 17, R: 19, B:14, T:22, A:10, F:26!!
rank = 18: xs = 150, ys = 0, zs = 100, mx = 300, my = 200, mz = 200!
rank = 18: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 18: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 1: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 3, NbrRanks are: L: 2, R: 4, B:-1, T:7, A:-5, F:11!!
rank = 3: xs = 225, ys = 0, zs = 0, mx = 300, my = 200, mz = 200!
rank = 3: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0
rank = 3: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 9, NbrRanks are: L: 8, R: 10, B:5, T:13, A:1, F:17!!
rank = 9: xs = 75, ys = 0, zs = 50, mx = 300, my = 200, mz = 200!
rank = 9: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 9: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 23, NbrRanks are: L: 22, R: 24, B:19, T:27, A:15, F:31!!
rank = 23: xs = 225, ys = 100, zs = 100, mx = 300, my = 200, mz = 200!
rank = 23: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 23: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 15, NbrRanks are: L: 14, R: 16, B:11, T:19, A:7, F:23!!
rank = 15: xs = 225, ys = 100, zs = 50, mx = 300, my = 200, mz = 200!
rank = 15: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 15: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 11, NbrRanks are: L: 10, R: 12, B:7, T:15, A:3, F:19!!
rank = 11: xs = 225, ys = 0, zs = 50, mx = 300, my = 200, mz = 200!
rank = 11: left BC = 0, rightBC = 1, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 11: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 8, NbrRanks are: L: 7, R: 9, B:4, T:12, A:0, F:16!!
rank = 8: xs = 0, ys = 0, zs = 50, mx = 300, my = 200, mz = 200!
rank = 8: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 8: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 13, NbrRanks are: L: 12, R: 14, B:9, T:17, A:5, F:21!!
rank = 13: xs = 75, ys = 100, zs = 50, mx = 300, my = 200, mz = 200!
rank = 13: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 14, NbrRanks are: L: 13, R: 15, B:10, T:18, A:6, F:22!!
rank = 14: xs = 150, ys = 100, zs = 50, mx = 300, my = 200, mz = 200!
rank = 14: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 14: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 13: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
Ready for Initialization!!
node = 0 mx = 300 my = 200 mz = 200 mm = 4 nn = 2 pp = 4
rank = 0, NbrRanks are: L: -1, R: 1, B:-4, T:4, A:-8, F:8!!
rank = 0: xs = 0, ys = 0, zs = 0, mx = 300, my = 200, mz = 200!
rank = 0: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0
rank = 0: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 7, NbrRanks are: L: 6, R: 8, B:3, T:11, A:-1, F:15!!
rank = 7: xs = 225, ys = 100, zs = 0, mx = 300, my = 200, mz = 200!
rank = 7: left BC = 0, rightBC = 1, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0
rank = 7: xc = 74, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 22, NbrRanks are: L: 21, R: 23, B:18, T:26, A:14, F:30!!
rank = 22: xs = 150, ys = 100, zs = 100, mx = 300, my = 200, mz = 200!
rank = 22: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 22: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 16, NbrRanks are: L: 15, R: 17, B:12, T:20, A:8, F:24!!
rank = 16: xs = 0, ys = 0, zs = 100, mx = 300, my = 200, mz = 200!
rank = 16: left BC = 1, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 16: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 5, NbrRanks are: L: 4, R: 6, B:1, T:9, A:-3, F:13!!
rank = 5: xs = 75, ys = 100, zs = 0, mx = 300, my = 200, mz = 200!
rank = 5: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0
rank = 5: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 6, NbrRanks are: L: 5, R: 7, B:2, T:10, A:-2, F:14!!
rank = 6: xs = 150, ys = 100, zs = 0, mx = 300, my = 200, mz = 200!
rank = 6: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 1, front BC = 0
rank = 6: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 2, NbrRanks are: L: 1, R: 3, B:-2, T:6, A:-6, F:10!!
rank = 2: xs = 150, ys = 0, zs = 0, mx = 300, my = 200, mz = 200!
rank = 2: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 1, front BC = 0
rank = 10, NbrRanks are: L: 9, R: 11, B:6, T:14, A:2, F:18!!
rank = 10: xs = 150, ys = 0, zs = 50, mx = 300, my = 200, mz = 200!
rank = 2: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 10: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 10: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 17, NbrRanks are: L: 16, R: 18, B:13, T:21, A:9, F:25!!
rank = 17: xs = 75, ys = 0, zs = 100, mx = 300, my = 200, mz = 200!
rank = 17: left BC = 0, rightBC = 0, bottom BC = 1, top BC = 0, aft BC = 0, front BC = 0
rank = 17: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank = 21, NbrRanks are: L: 20, R: 22, B:17, T:25, A:13, F:29!!
rank = 21: xs = 75, ys = 100, zs = 100, mx = 300, my = 200, mz = 200!
rank = 21: left BC = 0, rightBC = 0, bottom BC = 0, top BC = 1, aft BC = 0, front BC = 0
rank = 21: xc = 75, yc = 100, zc = 50, xw = 75, yw = 100, zw = 50
rank 8 Cylinder: neighbor = left: 59 right: 75 bottom: 49 top: 100 aft: 49 front: 100
rank 9 Cylinder: neighbor = left: 74 right: 150 bottom: 49 top: 100 aft: 49 front: 100
rank 11 Cylinder: neighbor = left: 224 right: 161 bottom: 49 top: 100 aft: 49 front: 100
rank 14 Cylinder: neighbor = left: 149 right: 161 bottom: 99 top: 151 aft: 49 front: 100
rank 12 Cylinder: neighbor = left: 59 right: 75 bottom: 99 top: 151 aft: 49 front: 100
rank 10 Cylinder: neighbor = left: 149 right: 161 bottom: 49 top: 100 aft: 49 front: 100
rank 13 Cylinder: neighbor = left: 74 right: 150 bottom: 99 top: 151 aft: 49 front: 100
rank 15 Cylinder: neighbor = left: 224 right: 161 bottom: 99 top: 151 aft: 49 front: 100
rank 1 Cylinder: neighbor = left: 74 right: 150 bottom: 49 top: 100 aft: 49 front: 50
Center of the sphere = (8.635204, 12.139085, 12.139085) !!
rank 17 Cylinder: neighbor = left: 74 right: 150 bottom: 49 top: 100 aft: 99 front: 150
rank 27 Cylinder: neighbor = left: 224 right: 161 bottom: 49 top: 100 aft: 149 front: 151
rank 24 Cylinder: neighbor = left: 59 right: 75 bottom: 49 top: 100 aft: 149 front: 151
rank 26 Cylinder: neighbor = left: 149 right: 161 bottom: 49 top: 100 aft: 149 front: 151
rank 31 Cylinder: neighbor = left: 224 right: 161 bottom: 99 top: 151 aft: 149 front: 151
rank 28 Cylinder: neighbor = left: 59 right: 75 bottom: 99 top: 151 aft: 149 front: 151
rank 19 Cylinder: neighbor = left: 224 right: 161 bottom: 49 top: 100 aft: 99 front: 150
rank 21 Cylinder: neighbor = left: 74 right: 150 bottom: 99 top: 151 aft: 99 front: 150
rank 18 Cylinder: neighbor = left: 149 right: 161 bottom: 49 top: 100 aft: 99 front: 150
rank 20 Cylinder: neighbor = left: 59 right: 75 bottom: 99 top: 151 aft: 99 front: 150
rank 23 Cylinder: neighbor = left: 224 right: 161 bottom: 99 top: 151 aft: 99 front: 150
rank 30 Cylinder: neighbor = left: 149 right: 161 bottom: 99 top: 151 aft: 149 front: 151
rank 16 Cylinder: neighbor = left: 59 right: 75 bottom: 49 top: 100 aft: 99 front: 150
rank 5 Cylinder: neighbor = left: 74 right: 150 bottom: 99 top: 151 aft: 49 front: 50
rank 7 Cylinder: neighbor = left: 224 right: 161 bottom: 99 top: 151 aft: 49 front: 50
rank 2 Cylinder: neighbor = left: 149 right: 161 bottom: 49 top: 100 aft: 49 front: 50
rank 3 Cylinder: neighbor = left: 224 right: 161 bottom: 49 top: 100 aft: 49 front: 50
rank 0 Cylinder: neighbor = left: 59 right: 75 bottom: 49 top: 100 aft: 49 front: 50
rank 4 Cylinder: neighbor = left: 59 right: 75 bottom: 99 top: 151 aft: 49 front: 50
rank 6 Cylinder: neighbor = left: 149 right: 161 bottom: 99 top: 151 aft: 49 front: 50
rank 25 Cylinder: neighbor = left: 74 right: 150 bottom: 49 top: 100 aft: 149 front: 151
rank 22 Cylinder: neighbor = left: 149 right: 161 bottom: 99 top: 151 aft: 99 front: 150
rank 29 Cylinder: neighbor = left: 74 right: 150 bottom: 99 top: 151 aft: 149 front: 151
Ready for the main iteration!!
Current time step = 0, Total time step number = 1
Check ApplyBCs!!
Check CalConvTerm!!
Check CommUpdate after CalConvTerm!
Check CalIBVel!
Check CalIBForce!!
Check AssyForce!!
Check Finished AssyForce!!
Ready for ApplyNonUniform of the KSPSolve#1!!
Ready for KSPSolve#1!!
0 KSP Residual norm 2.997792736227e+02
1 KSP Residual norm 1.591262605100e+01
2 KSP Residual norm 2.660865065368e+00
3 KSP Residual norm 5.661457826860e-01
4 KSP Residual norm 6.273811571672e-02
5 KSP Residual norm 8.541664768857e-03
6 KSP Residual norm 1.196432453759e-03
7 KSP Residual norm 1.366852070104e-04
8 KSP Residual norm 4.984885606812e-05
9 KSP Residual norm 2.967025150410e-05
KSP Object: 32 MPI processes
type: gmres
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
GMRES: happy breakdown tolerance 1e-30
maximum iterations=10000
tolerances: relative=1e-07, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using PRECONDITIONED norm type for convergence test
PC Object: 32 MPI processes
type: hypre
HYPRE BoomerAMG preconditioning
HYPRE BoomerAMG: Cycle type V
HYPRE BoomerAMG: Maximum number of levels 25
HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
HYPRE BoomerAMG: Threshold for strong coupling 0.25
HYPRE BoomerAMG: Interpolation truncation factor 0
HYPRE BoomerAMG: Interpolation: max elements per row 0
HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
HYPRE BoomerAMG: Maximum row sums 0.9
HYPRE BoomerAMG: Sweeps down 1
HYPRE BoomerAMG: Sweeps up 1
HYPRE BoomerAMG: Sweeps on coarse 1
HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi
HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi
HYPRE BoomerAMG: Relax on coarse Gaussian-elimination
HYPRE BoomerAMG: Relax weight (all) 1
HYPRE BoomerAMG: Outer relax weight (all) 1
HYPRE BoomerAMG: Using CF-relaxation
HYPRE BoomerAMG: Measure type local
HYPRE BoomerAMG: Coarsen type Falgout
HYPRE BoomerAMG: Interpolation type classical
linear system matrix = precond matrix:
Matrix Object: 32 MPI processes
type: mpiaij
rows=12000000, cols=12000000
Solver#1, Rank = 1, Solve One Time = 165.783827
total: nonzeros=83680000, allocated nonzeros=83680000
Solver#1, Rank = 8, Solve One Time = 165.783472
Solver#1, Rank = 12, Solve One Time = 165.783675
Solver#1, Rank = 10, Solve One Time = 165.773235
Solver#1, Rank = 2, Solve One Time = 165.783872
Solver#1, Rank = 13, Solve One Time = 165.783859
Solver#1, Rank = 9, Solve One Time = 165.783889
total number of mallocs used during MatSetValues calls =0
Solver#1, Rank = 6, Solve One Time = 165.782438
Solver#1, Rank = 0, Solve One Time = 165.783631
Solver#1, Rank = 3, Solve One Time = 165.782258
Solver#1, Rank = 7, Solve One Time = 165.782349
Solver#1, Rank = 24, Solve One Time = 165.783867
Finish KSPSolve#1!!
Solver#1, Rank = 11, Solve One Time = 165.773358
Solver#1, Rank = 16, Solve One Time = 165.783255
Solver#1, Rank = 25, Solve One Time = 165.784052
Solver#1, Rank = 4, Solve One Time = 165.783982
Solver#1, Rank = 14, Solve One Time = 165.781645
Solver#1, Rank = 26, Solve One Time = 165.784008
Solver#1, Rank = 28, Solve One Time = 165.784051
Solver#1, Rank = 5, Solve One Time = 165.782255
Solver#1, Rank = 15, Solve One Time = 165.781732
Solver#1, Rank = 18, Solve One Time = 165.776823
Solver#1, Rank = 27, Solve One Time = 165.773497
Solver#1, Rank = 17, Solve One Time = 165.783936
Solver#1, Rank = 29, Solve One Time = 165.784046
Solver#1, Rank = 20, Solve One Time = 165.784034
Solver#1, Rank = 19, Solve One Time = 165.771977
Solver#1, Rank = 30, Solve One Time = 165.773829
Solver#1, Rank = 21, Solve One Time = 165.784062
Solver#1, Rank = 31, Solve One Time = 165.773872
Solver#1, Rank = 22, Solve One Time = 165.766045
Solver#1, Rank = 23, Solve One Time = 165.766240
Residual norm 2.05135e-07
Check#2 ApplyBCs!!
Check#2 AssyForce!!
Check#2 CalVelField!!
0 KSP Residual norm 2.870247935899e+02
1 KSP Residual norm 1.606442422016e+01
2 KSP Residual norm 3.090064201333e+00
3 KSP Residual norm 5.867933774996e-01
4 KSP Residual norm 7.308417770262e-02
5 KSP Residual norm 1.103028493864e-02
6 KSP Residual norm 2.170332527761e-03
7 KSP Residual norm 6.906994551513e-04
8 KSP Residual norm 3.045998288429e-04
9 KSP Residual norm 5.136309989396e-05
10 KSP Residual norm 2.055236160250e-05
KSP Object: 32 MPI processes
type: gmres
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
GMRES: happy breakdown tolerance 1e-30
maximum iterations=10000
tolerances: relative=1e-07, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using PRECONDITIONED norm type for convergence test
PC Object: 32 MPI processes
type: hypre
HYPRE BoomerAMG preconditioning
HYPRE BoomerAMG: Cycle type V
HYPRE BoomerAMG: Maximum number of levels 25
HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
HYPRE BoomerAMG: Threshold for strong coupling 0.25
HYPRE BoomerAMG: Interpolation truncation factor 0
HYPRE BoomerAMG: Interpolation: max elements per row 0
HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
HYPRE BoomerAMG: Maximum row sums 0.9
HYPRE BoomerAMG: Sweeps down 1
HYPRE BoomerAMG: Sweeps up 1
HYPRE BoomerAMG: Sweeps on coarse 1
HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi
HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi
HYPRE BoomerAMG: Relax on coarse Gaussian-elimination
HYPRE BoomerAMG: Relax weight (all) 1
HYPRE BoomerAMG: Outer relax weight (all) 1
HYPRE BoomerAMG: Using CF-relaxation
HYPRE BoomerAMG: Measure type local
HYPRE BoomerAMG: Coarsen type Falgout
HYPRE BoomerAMG: Interpolation type classical
linear system matrix = precond matrix:
Matrix Object: 32 MPI processes
type: mpiaij
rows=12000000, cols=12000000
Solver#2, Rank = 8, Solve One Time = 171.588158
total: nonzeros=83680000, allocated nonzeros=83680000
Solver#2, Rank = 1, Solve One Time = 171.588110
Solver#2, Rank = 9, Solve One Time = 171.586617
Solver#2, Rank = 4, Solve One Time = 171.588193
Solver#2, Rank = 14, Solve One Time = 171.588117
Solver#2, Rank = 15, Solve One Time = 171.587124
Solver#2, Rank = 24, Solve One Time = 171.588076
Solver#2, Rank = 10, Solve One Time = 171.586486
total number of mallocs used during MatSetValues calls =0
Solver#2, Rank = 11, Solve One Time = 171.586354
Solver#2, Rank = 26, Solve One Time = 171.588102
Solver#2, Rank = 5, Solve One Time = 171.582924
Solver#2, Rank = 13, Solve One Time = 171.588282
Solver#2, Rank = 28, Solve One Time = 171.588074
Solver#2, Rank = 0, Solve One Time = 171.588347
Solver#2, Rank = 12, Solve One Time = 171.588397
Solver#2, Rank = 25, Solve One Time = 171.588135
Solver#2, Rank = 6, Solve One Time = 171.575956
Solver#2, Rank = 3, Solve One Time = 171.578867
Solver#2, Rank = 2, Solve One Time = 171.588348
Solver#2, Rank = 7, Solve One Time = 171.575976
Solver#2, Rank = 16, Solve One Time = 171.588326
Solver#2, Rank = 27, Solve One Time = 171.580220
Solver#2, Rank = 17, Solve One Time = 171.588328
Solver#2, Rank = 29, Solve One Time = 171.583465
Solver#2, Rank = 18, Solve One Time = 171.588341
Solver#2, Rank = 30, Solve One Time = 171.582318
Solver#2, Rank = 20, Solve One Time = 171.588311
Solver#2, Rank = 31, Solve One Time = 171.582314
Solver#2, Rank = 19, Solve One Time = 171.586152
Solver#2, Rank = 21, Solve One Time = 171.586144
Solver#2, Rank = 22, Solve One Time = 171.586817
Solver#2, Rank = 23, Solve One Time = 171.586864
Residual norm 9.57625e-08
Rank#18, Max dp = 1.231050e+00 @ (8, 81, 4)
Rank#13, Re = 300.000000, time step = 0, continuity = 2.612951e-01 @ Phys(107, 150, 96) & Local(36, 54, 50) with (0.010000, 0.010000, 0.010000), (0.839689,0.625750,-0.110498,0.055790,-0.011703,0.033334)
Restart files have been written!!
PlotU is ready!!
PlotU is finished!!
More information about the petsc-users
mailing list