<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>Hello Mark,</div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Actually I have tried those options for a sequential run where I need to use superlu to get the condition number of some matrix.</div></span></span>
</div><div><br class="webkit-block-placeholder"></div><div>In the dev version,</div><div><br></div><div><div>ierr = DMCreateMatrix(DMMGGetDM(dmmg), MATAIJ, &jacobian);CHKERRQ(ierr);</div></div><div><div>ierr = MatSetOption(jacobian, MAT_IGNORE_ZERO_ENTRIES, PETSC_TRUE);CHKERRQ(ierr);</div></div><div><br></div><div>And in the options file, </div><div><br></div><div>-dm_preallocate_only</div><div><br></div><div>is added.</div><div><br></div><div>This is totally fine when np=1, however, when I use multiple processors, there are some memory corruption happened.</div><div><br></div><div>For example, the number of true nonzeros for 65536 size matrix is 1,470,802. the output (&) is for np=1 with the following PETSc related options:</div><div><div><br></div><div>-dm_preallocate_only</div><div><div>-snes_ksp_ew true</div><div>-snes_monitor</div><div>-snes_max_it 1</div><div>-ksp_view</div><div>-mat_view_info</div></div><div><div>-ksp_type preonly</div><div>-pc_type lu</div></div></div><div><div>-pc_factor_mat_solver_package superlu</div><div>-mat_superlu_conditionnumber</div><div>-mat_superlu_printstat</div></div><div><br></div><div><br></div><div>However, when np=2, the number of nonzeros changes to 3,366,976 with the following PETSc related options. (*) is the output file.</div><div><br></div><div>-dm_preallocate_only</div><div><div>-snes_ksp_ew true</div><div>-snes_monitor</div><div>-ksp_view</div><div>-mat_view_info</div></div><div><div>-ksp_type preonly</div><div>-pc_type lu</div><div>-pc_factor_mat_solver_package superlu_dist</div></div><div><br></div><div>-----------------------------</div><div>(&)</div><div><br></div><div><div>*************petsc-Dev = yes*****************</div><div>*********************************************</div><div>******* start solving for time = 1.00000 at time step = 1******</div><div> 0 SNES Function norm 1.242539468950e-02</div><div>Matrix Object: 1 MPI processes</div><div> type: seqaij</div><div> rows=65536, cols=65536</div><div> total: nonzeros=1470802, allocated nonzeros=2334720</div><div> total number of mallocs used during MatSetValues calls =0</div><div> not using I-node routines</div><div> Recip. condition number = 4.345658e-07</div><div>MatLUFactorNumeric_SuperLU():</div><div>Factor time = 42.45</div><div>Factor flops = 7.374620e+10 Mflops = 1737.25</div><div>Solve time = 0.00</div><div>Number of memory expansions: 3</div><div> No of nonzeros in factor L = 32491856</div><div> No of nonzeros in factor U = 39390974</div><div> No of nonzeros in L+U = 71817294</div><div> L\U MB 741.397 total MB needed 756.339</div><div>Matrix Object: 1 MPI processes</div><div> type: seqaij</div></div><div><div> rows=65536, cols=65536</div><div> package used to perform factorization: superlu</div><div> total: nonzeros=0, allocated nonzeros=0</div><div> total number of mallocs used during MatSetValues calls =0</div><div> SuperLU run parameters:</div><div> Equil: NO</div><div> ColPerm: 3</div><div> IterRefine: 0</div><div> SymmetricMode: NO</div><div> DiagPivotThresh: 1</div><div> PivotGrowth: NO</div><div> ConditionNumber: YES</div><div> RowPerm: 0</div><div> ReplaceTinyPivot: NO</div><div> PrintStat: YES</div><div> lwork: 0</div><div>MatSolve__SuperLU():</div><div>Factor time = 42.45</div><div>Factor flops = 7.374620e+10 Mflops = 1737.25</div><div>Solve time = 0.59</div><div>Solve flops = 1.436365e+08 Mflops = 243.45</div></div><div><div>Number of memory expansions: 3</div></div><div><div> 1 SNES Function norm 2.645145585949e-04</div></div><div><br></div><div><br></div><div>-----------------------------------------------</div><div>(*)</div><div><div>*************petsc-Dev = yes*****************</div><div>*********************************************</div><div>******* start solving for time = 1.00000 at time step = 1******</div><div> 0 SNES Function norm 1.242539468950e-02</div><div>Matrix Object: 2 MPI processes</div><div> type: mpiaij</div><div> rows=65536, cols=65536</div><div> total: nonzeros=3366976, allocated nonzeros=6431296</div><div> total number of mallocs used during MatSetValues calls =0</div><div> Matrix Object: 2 MPI processes</div><div> type: mpiaij</div><div> rows=65536, cols=65536</div><div> total: nonzeros=3366976, allocated nonzeros=3366976</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node (on process 0) routines: found 8192 nodes, limit used is 5</div><div>Input Error: Incorrect objective type.</div><div>Input Error: Incorrect objective type.</div></div><div><div>At column 0, pivotL() encounters zero diagonal at line 708 in file symbfact.c</div><div>At column 0, pivotL() encounters zero diagonal at line 708 in file symbfact.c</div></div><div><br></div><div>Moreover, When I use valgrind with --leak-check=yes --track-origins=yes, there are 441 errors from 219 contexts in PetscInitialize() before calling SNESSolve(). Is this normal for dev?</div><div><br></div><div>Thanks very much!</div><div><br></div><div>Best regards,</div><div><br></div><div>Rebecca</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div>On Jan 26, 2012, at 5:06 PM, Jed Brown wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div class="gmail_quote">On Thu, Jan 26, 2012 at 19:00, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm guessing that PETSc recently changed and now filters out 0.0 in MatSetValues ... is this true?<br></blockquote><div><br></div><div>Did the option MAT_IGNORE_ZERO_ENTRIES get set somehow?</div></div>
</blockquote></div><br></div></body></html>