Hi Barry,<br><br>Thank you for your reply.<br>I don't think this problem comes from the matrix assemble. Because the result I showed you in the last email is from a two-level Newton method which means I first solve a coarse problem and use the coarse solution as the fine level problem's initial guess. If I just use the one-level method, there is no such problem. The memory usage in the -log_summary output is correct and time spend on the SNESJacobianEval is also normal I think (see attached) for the one-level method. The strange memory usage just appear in the two-level method. The reason that I claim the two-level's computing time is not correct is that I solve the same problem with the same number of processors and the two-level's iteration number of SNES and GMRES is much smaller than the one-level method, but the compute time is opposite (the time spend on the coarse problem is just 25s). From the -log_summary outputs of the two methods I found that the matrix's memory usage is total different. So I think there must be some bugs in my two-level code. But I have no idea how to debug this problem. <br>
<br>Best,<br>Rongliang<br><br><div class="gmail_quote">On Fri, Oct 7, 2011 at 10:24 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly</a><br>
<br>
<br>
On Oct 7, 2011, at 11:22 AM, Rongliang Chen wrote:<br>
<br>
> -------------------------------------------------<br>
> Joab<br>
><br>
> Shape Optimization solver<br>
> by Rongliang Chen<br>
> compiled on 15:54:32, Oct 3 2011<br>
> Running on: Wed Oct 5 10:24:10 2011<br>
><br>
> revision $Rev: 157 $<br>
> -------------------------------------------------<br>
> Command-line options: -coarse_ksp_rtol 1.0e-1 -coarsegrid /scratch/stmp00/ronglian/input/Cannula/Cannula_Nest2_E2000_N8241_D70170.fsi -computeinitialguess -f /scratch/stmp00/ronglian/input/Cannula/Cannula_Nest2_E32000_N128961_D1096650.fsi -geometric_asm -geometric_asm_overlap 8 -inletu 5.0 -ksp_atol 1e-8 -ksp_gmres_restart 600 -ksp_max_it 3000 -ksp_pc_side right -ksp_rtol 1.e-3 -ksp_type gmres -log_summary -mat_partitioning_type parmetis -nest_geometric_asm_overlap 4 -nest_ksp_atol 1e-8 -nest_ksp_gmres_restart 800 -nest_ksp_max_it 1000 -nest_ksp_pc_side right -nest_ksp_rtol 1.e-2 -nest_ksp_type gmres -nest_pc_asm_type basic -nest_pc_type asm -nest_snes_atol 1.e-10 -nest_snes_max_it 20 -nest_snes_rtol 1.e-4 -nest_sub_pc_factor_mat_ordering_type qmd -nest_sub_pc_factor_shift_amount 1e-8 -nest_sub_pc_factor_shift_type nonzero -nest_sub_pc_type lu -nested -noboundaryreduce -pc_asm_type basic -pc_type asm -shapebeta 10.0 -snes_atol 1.e-10 -snes_max_it 20 -snes_rtol 1.e-6 -sub_pc_f<br>
> actor_mat_ordering_type qmd -sub_pc_factor_shift_amount 1e-8 -sub_pc_factor_shift_type nonzero -sub_pc_type lu -viscosity 0.01<br>
> -------------------------------------------------<br>
><br>
> Starting to load grid...<br>
> Nodes on moving boundary: coarse 199, fine 799, Gridratio 0.250000.<br>
> Setupping Interpolation matrix......<br>
> Interpolation matrix done......Time spent: 0.405431<br>
> finished.<br>
> Grid has 32000 elements, 1096658 degrees of freedom.<br>
> Coarse grid has 2000 elements, 70170 degrees of freedom.<br>
> [0] has 35380 degrees of freedom (matrix), 35380 degrees of freedom (including shared points).<br>
> [0] coarse grid has 2194 degrees of freedom (matrix), 2194 degrees of freedom (including shared points).<br>
> [31] has 32466 degrees of freedom (matrix), 34428 degrees of freedom (including shared points).<br>
> [31] coarse grid has 2250 degrees of freedom (matrix), 2826 degrees of freedom (including shared points).<br>
> Time spend on the load grid and create matrix etc.: 3.577862.<br>
> Solving fixed mesh (steady-state problem)<br>
> Solving coarse problem......<br>
> 0 SNES norm 3.1224989992e+01, 0 KSP its last norm 0.0000000000e+00.<br>
> 1 SNES norm 1.3987219837e+00, 25 KSP its last norm 2.4915963656e-01.<br>
> 2 SNES norm 5.1898321541e-01, 59 KSP its last norm 1.3451744761e-02.<br>
> 3 SNES norm 4.0024228221e-02, 56 KSP its last norm 4.9036146089e-03.<br>
> 4 SNES norm 6.7641787439e-04, 59 KSP its last norm 3.6925683196e-04.<br>
> Coarse solver done......<br>
> Initial value of object function (Energy dissipation) (Coarse): 38.9341108701<br>
> 0 SNES norm 7.4575110699e+00, 0 KSP its last norm 0.0000000000e+00.<br>
> 1 SNES norm 6.4497565921e-02, 51 KSP its last norm 7.4277453141e-03.<br>
> 2 SNES norm 9.2093642958e-04, 90 KSP its last norm 5.4331380112e-05.<br>
> 3 SNES norm 8.1283574549e-07, 103 KSP its last norm 7.5974191049e-07.<br>
> Initial value of object function (Energy dissipation) (Fine): 42.5134271399<br>
> Solution time of 17.180358 sec.<br>
> Fixed mesh (Steady-state) solver done.<br>
> Total number of nonlinear iterations = 3<br>
> Total number of linear iterations = 244<br>
> Average number of linear iterations = 81.333336<br>
> Time computing: 17.180358 sec, Time outputting: 0.000000 sec.<br>
> Time spent in coarse nonlinear solve: 0.793436 sec, 0.046183 fraction of total compute time.<br>
> Solving Shape Optimization problem (steady-state problem)<br>
> Solving coarse problem......<br>
> 0 SNES norm 4.1963166116e+01, 0 KSP its last norm 0.0000000000e+00.<br>
> 1 SNES norm 3.2749386875e+01, 132 KSP its last norm 4.0966334477e-01.<br>
> 2 SNES norm 2.2874504408e+01, 130 KSP its last norm 3.2526355310e-01.<br>
> 3 SNES norm 1.4327187891e+01, 132 KSP its last norm 2.1213029400e-01.<br>
> 4 SNES norm 1.7283643754e+00, 81 KSP its last norm 1.4233338128e-01.<br>
> 5 SNES norm 3.6703566918e-01, 133 KSP its last norm 1.6069896349e-02.<br>
> 6 SNES norm 3.6554528686e-03, 77 KSP its last norm 3.5379167356e-03.<br>
> Coarse solver done......<br>
> Optimized value of object function (Energy dissipation) (Coarse): 29.9743062939<br>
> The reduction of the energy dissipation (Coarse): 23.012737%<br>
> The optimized curve (Coarse):<br>
> a = (4.500000, -0.042893, -0.002030, 0.043721, -0.018798, 0.001824)<br>
> Solving moving mesh equation......<br>
> KSP norm 2.3040219081e-07, KSP its. 741. Time spent 8.481956<br>
> Moving mesh solver done.<br>
> 0 SNES norm 4.7843968670e+02, 0 KSP its last norm 0.0000000000e+00.<br>
> 1 SNES norm 1.0148854085e+02, 49 KSP its last norm 4.7373180511e-01.<br>
> 2 SNES norm 1.8312214030e+00, 46 KSP its last norm 1.0133332840e-01.<br>
> 3 SNES norm 3.3101970861e-03, 212 KSP its last norm 1.7753271069e-03.<br>
> 4 SNES norm 4.9552614008e-06, 249 KSP its last norm 3.2293284103e-06.<br>
> Optimized value of object function (Energy dissipation) (Fine): 33.2754372645<br>
> Solution time of 4053.227456 sec.<br>
> Number of unknowns = 1096658<br>
> Parameters: kinematic viscosity = 0.01<br>
> inlet velocity: u = 5, v = 0<br>
> Total number of nonlinear iterations = 4<br>
> Total number of linear iterations = 556<br>
> Average number of linear iterations = 139.000000<br>
> Time computing: 4053.227456 sec, Time outputting: 0.000001 sec.<br>
> Time spent in coarse nonlinear solve: 24.239526 sec, 0.005980 fraction of total compute time.<br>
> The optimized curve (fine):<br>
> a = (4.500000, -0.046468, -0.001963, 0.045736, -0.019141, 0.001789)<br>
> The reduction of the energy dissipation (Fine): 21.729582%<br>
> Time spend on fixed mesh solving: 17.296872<br>
> Time spend on shape opt. solving: 4053.250126<br>
> Latex command line:<br>
> np Newton GMRES Time(Total) Time(Coarse) Ratio<br>
> 32 & 4 & 139.00 & 4053.23 & 24.24 & 0.6\%<br>
><br>
> Running finished on: Wed Oct 5 11:32:04 2011<br>
> Total running time: 4070.644329<br>
> ************************************************************************************************************************<br>
> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***<br>
> ************************************************************************************************************************<br>
><br>
> ---------------------------------------------- PETSc Performance Summary: ----------------------------------------------<br>
><br>
> ./joab on a Janus-nod named node1751 with 32 processors, by ronglian Wed Oct 5 11:32:04 2011<br>
> Using Petsc Release Version 3.2.0, Patch 1, Mon Sep 12 16:01:51 CDT 2011<br>
><br>
> Max Max/Min Avg Total<br>
> Time (sec): 4.074e+03 1.00000 4.074e+03<br>
> Objects: 1.011e+03 1.00000 1.011e+03<br>
> Flops: 2.255e+11 2.27275 1.471e+11 4.706e+12<br>
> Flops/sec: 5.535e+07 2.27275 3.609e+07 1.155e+09<br>
> MPI Messages: 1.103e+05 5.41392 3.665e+04 1.173e+06<br>
> MPI Message Lengths: 1.326e+09 2.60531 2.416e+04 2.833e+10<br>
> MPI Reductions: 5.969e+03 1.00000<br>
><br>
> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)<br>
> e.g., VecAXPY() for real vectors of length N --> 2N flops<br>
> and VecAXPY() for complex vectors of length N --> 8N flops<br>
><br>
> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- -- Message Lengths -- -- Reductions --<br>
> Avg %Total Avg %Total counts %Total Avg %Total counts %Total<br>
> 0: Main Stage: 4.0743e+03 100.0% 4.7058e+12 100.0% 1.173e+06 100.0% 2.416e+04 100.0% 5.968e+03 100.0%<br>
><br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> See the 'Profiling' chapter of the users' manual for details on interpreting output.<br>
> Phase summary info:<br>
> Count: number of times phase was executed<br>
> Time and Flops: Max - maximum over all processors<br>
> Ratio - ratio of maximum to minimum over all processors<br>
> Mess: number of messages sent<br>
> Avg. len: average message length<br>
> Reduct: number of global reductions<br>
> Global: entire computation<br>
> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().<br>
> %T - percent time in this phase %F - percent flops in this phase<br>
> %M - percent messages in this phase %L - percent message lengths in this phase<br>
> %R - percent reductions in this phase<br>
> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
> Event Count Time (sec) Flops --- Global --- --- Stage --- Total<br>
> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
><br>
> --- Event Stage 0: Main Stage<br>
><br>
> MatMult 2493 1.0 1.2225e+0218.4 4.37e+09 1.1 3.9e+05 2.2e+03 0.0e+00 2 3 33 3 0 2 3 33 3 0 1084<br>
> MatMultTranspose 6 1.0 3.3590e-02 2.2 7.38e+06 1.1 8.0e+02 1.5e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 6727<br>
> MatSolve 2467 1.0 1.1270e+02 1.7 5.95e+10 1.7 0.0e+00 0.0e+00 0.0e+00 2 33 0 0 0 2 33 0 0 0 13775<br>
> MatLUFactorSym 4 1.0 3.4774e+00 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 0 0<br>
> MatLUFactorNum 18 1.0 2.0832e+02 3.7 1.55e+11 3.2 0.0e+00 0.0e+00 0.0e+00 2 56 0 0 0 2 56 0 0 0 12746<br>
> MatILUFactorSym 1 1.0 8.3280e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatAssemblyBegin 103 1.0 7.6879e+0215.4 0.00e+00 0.0 1.6e+04 6.2e+04 1.7e+02 7 0 1 4 3 7 0 1 4 3 0<br>
> MatAssemblyEnd 103 1.0 3.7818e+01 1.0 0.00e+00 0.0 3.0e+03 5.3e+02 1.6e+02 1 0 0 0 3 1 0 0 0 3 0<br>
> MatGetRowIJ 5 1.0 4.8716e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatGetSubMatrice 18 1.0 4.3095e+00 2.5 0.00e+00 0.0 1.6e+04 3.5e+05 7.4e+01 0 0 1 20 1 0 0 1 20 1 0<br>
> MatGetOrdering 5 1.0 1.4656e+00 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+01 0 0 0 0 0 0 0 0 0 0 0<br>
> MatPartitioning 1 1.0 1.4356e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatZeroEntries 42 1.0 2.0939e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecDot 17 1.0 1.2719e-02 6.8 5.47e+05 1.1 0.0e+00 0.0e+00 1.7e+01 0 0 0 0 0 0 0 0 0 0 1317<br>
> VecMDot 2425 1.0 1.7196e+01 2.2 5.82e+09 1.1 0.0e+00 0.0e+00 2.4e+03 0 4 0 0 41 0 4 0 0 41 10353<br>
> VecNorm 2503 1.0 2.7923e+00 3.4 1.18e+08 1.1 0.0e+00 0.0e+00 2.5e+03 0 0 0 0 42 0 0 0 0 42 1293<br>
> VecScale 2467 1.0 7.3112e-02 1.7 5.84e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 24453<br>
> VecCopy 153 1.0 1.1636e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecSet 5031 1.0 6.0423e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecAXPY 137 1.0 1.1462e-02 1.5 6.33e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16902<br>
> VecWAXPY 19 1.0 1.7784e-03 1.4 2.83e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4869<br>
> VecMAXPY 2467 1.0 8.5820e+00 1.3 5.93e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 21153<br>
> VecAssemblyBegin 69 1.0 1.0341e+0018.2 0.00e+00 0.0 4.9e+03 5.4e+02 2.1e+02 0 0 0 0 3 0 0 0 0 3 0<br>
> VecAssemblyEnd 69 1.0 2.4939e-04 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecScatterBegin 7491 1.0 1.3734e+00 1.7 0.00e+00 0.0 1.1e+06 1.9e+04 0.0e+00 0 0 96 76 0 0 0 96 76 0 0<br>
> VecScatterEnd 7491 1.0 2.0055e+02 8.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0<br>
> VecReduceArith 8 1.0 1.4977e-03 2.0 3.05e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6232<br>
> VecReduceComm 4 1.0 8.9908e-0412.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecNormalize 2467 1.0 2.8067e+00 3.4 1.75e+08 1.1 0.0e+00 0.0e+00 2.4e+03 0 0 0 0 41 0 0 0 0 41 1905<br>
> SNESSolve 4 1.0 4.0619e+03 1.0 2.23e+11 2.3 9.4e+05 2.3e+04 4.1e+03100 98 80 77 68 100 98 80 77 68 1136<br>
> SNESLineSearch 17 1.0 1.1423e+01 1.0 5.23e+07 1.1 1.8e+04 1.7e+04 3.3e+02 0 0 2 1 6 0 0 2 1 6 140<br>
> SNESFunctionEval 23 1.0 2.9742e+01 1.0 2.60e+07 1.1 1.9e+04 1.9e+04 3.5e+02 1 0 2 1 6 1 0 2 1 6 27<br>
> SNESJacobianEval 17 1.0 3.6786e+03 1.0 0.00e+00 0.0 9.8e+03 6.4e+04 1.4e+02 90 0 1 2 2 90 0 1 2 2 0<br>
> KSPGMRESOrthog 2425 1.0 2.5150e+01 1.6 1.16e+10 1.1 0.0e+00 0.0e+00 2.4e+03 0 8 0 0 41 0 8 0 0 41 14157<br>
> KSPSetup 36 1.0 2.5388e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> KSPSolve 18 1.0 3.6141e+02 1.0 2.25e+11 2.3 1.1e+06 2.4e+04 5.0e+03 9100 97 96 84 9100 97 96 84 13015<br>
> PCSetUp 36 1.0 2.1635e+02 3.6 1.55e+11 3.2 1.8e+04 3.2e+05 1.5e+02 3 56 2 20 3 3 56 2 20 3 12274<br>
> PCSetUpOnBlocks 18 1.0 2.1293e+02 3.7 1.55e+11 3.2 0.0e+00 0.0e+00 2.7e+01 2 56 0 0 0 2 56 0 0 0 12471<br>
> PCApply 2467 1.0 2.5616e+02 2.5 5.95e+10 1.7 7.3e+05 2.8e+04 0.0e+00 4 33 62 73 0 4 33 62 73 0 6060<br>
> ------------------------------------------------------------------------------------------------------------------------<br>
><br>
> Memory usage is given in bytes:<br>
><br>
> Object Type Creations Destructions Memory Descendants' Mem.<br>
> Reports information only for process 0.<br>
><br>
> --- Event Stage 0: Main Stage<br>
><br>
> Matrix 39 39 18446744074642894848 0<br>
> Matrix Partitioning 1 1 640 0<br>
> Index Set 184 184 2589512 0<br>
> IS L to G Mapping 2 2 301720 0<br>
> Vector 729 729 133662888 0<br>
> Vector Scatter 29 29 30508 0<br>
> Application Order 2 2 9335968 0<br>
> SNES 4 4 5088 0<br>
> Krylov Solver 10 10 32264320 0<br>
> Preconditioner 10 10 9088 0<br>
> Viewer 1 0 0 0<br>
> ========================================================================================================================<br>
> Average time to get PetscTime(): 1.19209e-07<br>
> Average time for MPI_Barrier(): 1.20163e-05<br>
> Average time for zero size MPI_Send(): 2.49594e-06<br>
> #PETSc Option Table entries:<br>
> -coarse_ksp_rtol 1.0e-1<br>
> -coarsegrid /scratch/stmp00/ronglian/input/Cannula/Cannula_Nest2_E2000_N8241_D70170.fsi<br>
> -computeinitialguess<br>
> -f /scratch/stmp00/ronglian/input/Cannula/Cannula_Nest2_E32000_N128961_D1096650.fsi<br>
> -geometric_asm<br>
> -geometric_asm_overlap 8<br>
> -inletu 5.0<br>
> -ksp_atol 1e-8<br>
> -ksp_gmres_restart 600<br>
> -ksp_max_it 3000<br>
> -ksp_pc_side right<br>
> -ksp_rtol 1.e-3<br>
> -ksp_type gmres<br>
> -log_summary<br>
> -mat_partitioning_type parmetis<br>
> -nest_geometric_asm_overlap 4<br>
> -nest_ksp_atol 1e-8<br>
> -nest_ksp_gmres_restart 800<br>
> -nest_ksp_max_it 1000<br>
> -nest_ksp_pc_side right<br>
> -nest_ksp_rtol 1.e-2<br>
> -nest_ksp_type gmres<br>
> -nest_pc_asm_type basic<br>
> -nest_pc_type asm<br>
> -nest_snes_atol 1.e-10<br>
> -nest_snes_max_it 20<br>
> -nest_snes_rtol 1.e-4<br>
> -nest_sub_pc_factor_mat_ordering_type qmd<br>
> -nest_sub_pc_factor_shift_amount 1e-8<br>
> -nest_sub_pc_factor_shift_type nonzero<br>
> -nest_sub_pc_type lu<br>
> -nested<br>
> -noboundaryreduce<br>
> -pc_asm_type basic<br>
> -pc_type asm<br>
> -shapebeta 10.0<br>
> -snes_atol 1.e-10<br>
> -snes_max_it 20<br>
> -snes_rtol 1.e-6<br>
> -sub_pc_factor_mat_ordering_type qmd<br>
> -sub_pc_factor_shift_amount 1e-8<br>
> -sub_pc_factor_shift_type nonzero<br>
> -sub_pc_type lu<br>
> -viscosity 0.01<br>
> #End of PETSc Option Table entries<br>
> Compiled without FORTRAN kernels<br>
> Compiled with full precision matrices (default)<br>
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8<br>
> Configure run at: Tue Sep 13 13:28:48 2011<br>
> Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --with-batch=1 --with-mpi-shared-libraries=1 --known-mpi-shared-libraries=0 --download-f-blas-lapack=1 --download-hypre=1 --download-superlu=1 --download-parmetis=1 --download-superlu_dist=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --with-debugging=0<br>
> -----------------------------------------<br>
> Libraries compiled on Tue Sep 13 13:28:48 2011 on node1367<br>
> Machine characteristics: Linux-2.6.18-238.12.1.el5-x86_64-with-redhat-5.6-Tikanga<br>
> Using PETSc directory: /home/ronglian/soft/petsc-3.2-p1<br>
> Using PETSc arch: Janus-nodebug<br>
> -----------------------------------------<br>
><br>
> Using C compiler: mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O ${COPTFLAGS} ${CFLAGS}<br>
> Using Fortran compiler: mpif90 -Wall -Wno-unused-variable -O ${FOPTFLAGS} ${FFLAGS}<br>
> -----------------------------------------<br>
><br>
> Using include paths: -I/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/include -I/home/ronglian/soft/petsc-3.2-p1/include -I/home/ronglian/soft/petsc-3.2-p1/include -I/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/include -I/curc/tools/free/redhat_5_x86_64/openmpi-1.4.3_ib/include<br>
> -----------------------------------------<br>
><br>
> Using C linker: mpicc<br>
> Using Fortran linker: mpif90<br>
> Using libraries: -Wl,-rpath,/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/lib -L/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/lib -lpetsc -lX11 -Wl,-rpath,/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/lib -L/home/ronglian/soft/petsc-3.2-p1/Janus-nodebug/lib -lsuperlu_dist_2.5 -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lHYPRE -lmpi_cxx -lstdc++ -lscalapack -lblacs -lsuperlu_4.2 -lflapack -lfblas -L/curc/tools/free/redhat_5_x86_64/openmpi-1.4.3_ib/lib -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -lmpi_f90 -lmpi_f77 -lgfortran -lm -lm -lm -lm -lmpi_cxx -lstdc++ -lmpi_cxx -lstdc++ -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl<br>
> -----------------------------------------<br>
<div><div></div><div class="h5">><br>
>><br>
>> Yes, it has no influence on performance. If you think it does, send<br>
>> -log_summary output to <a href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a><br>
>><br>
>> Matt<br>
>><br>
>><br>
> Hi Matt,<br>
><br>
> The -log_summary output is attached. I found that the SNESJacobianEval()<br>
> takes 90% of the total time. I think this is abnormal because I use a hand<br>
> coded Jacobian matrix. The reason, I think, for the 90% of the total time is<br>
> that the matrix takes too much memory (over 1.8x10^19 bytes) which maybe<br>
> have used the swap. But I do not know why 23 one million by one million<br>
> matrices will use so much memory. Can you tell me how to debug this problem?<br>
> Thank you.<br>
><br>
> Best,<br>
> Rongliang<br>
><br>
><br>
> Yes, it has no influence on performance. If you think it does, send<br>
> -log_summary output to <a href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a><br>
><br>
> Matt<br>
><br>
><br>
> Hi Matt,<br>
><br>
> The -log_summary output is attached. I found that the SNESJacobianEval() takes 90% of the total time. I think this is abnormal because I use a hand coded Jacobian matrix. The reason, I think, for the 90% of the total time is that the matrix takes too much memory (over 1.8x10^19 bytes) which maybe have used the swap. But I do not know why 23 one million by one million matrices will use so much memory. Can you tell me how to debug this problem? Thank you.<br>
><br>
> Best,<br>
> Rongliang<br>
<br>
</div></div></blockquote></div><br>