Mark,<br><br>The changes pulled through this morning. I've run it with the options<br><br>-ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8<br>
<br>and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. <br>
<br>Any thoughts?<br><br>John<br><br><br><div class="gmail_quote">On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Humm, I see it with hg view (appended). </div><div><br></div><div>Satish, my main repo looks hosed. I see this:</div>
<div><br></div><div><div>~/Codes/petsc-dev>hg update</div><div class="im"><div>abort: crosses branches (merge branches or use --clean to discard changes)</div></div><div>~/Codes/petsc-dev>hg merge</div><div>abort: branch 'default' has 3 heads - please merge with an explicit rev</div>
<div>(run 'hg heads .' to see heads)</div><div>~/Codes/petsc-dev>hg heads</div><div>changeset: 22496:8e2a98268179</div><div>tag: tip</div><div>user: Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></div>
<div>date: Wed Mar 14 16:42:25 2012 -0500</div><div>files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c</div><div>description:</div><div>undoing manually changes I put in because Satish had a better fix</div>
<div><br></div><div><br></div><div>changeset: 22492:bda4df63072d</div><div>user: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></div><div>date: Wed Mar 14 17:39:52 2012 -0400</div>
<div>files: src/ksp/pc/impls/gamg/tools.c</div><div>description:</div><div>fix for unsymmetric matrices.</div><div><br></div><div><br></div><div>changeset: 22469:b063baf366e4</div><div>user: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></div>
<div>date: Wed Mar 14 14:22:28 2012 -0400</div><div>files: src/ksp/pc/impls/gamg/tools.c</div><div>description:</div><div>added fix for preallocation for unsymetric matrices.</div><div><br></div></div><div>Mark</div>
<div><br></div><div>my 'hg view' on my merge repo:</div><div><br></div><div>Revision: 22492</div><div>Branch: default</div><div>Author: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> 2012-03-14 17:39:52</div>
<div>Committer: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> 2012-03-14 17:39:52</div><div>Tags: tip</div><div>Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch)</div>
<div><br></div><div> fix for unsymmetric matrices.</div><div><br></div><div><br></div><div>------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------</div><div>@@ -103,7 +103,7 @@</div><div> PetscErrorCode ierr;</div>
<div> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc;</div><div> PetscMPIInt mype, npe;</div><div>- Mat Gmat = *a_Gmat, tGmat;</div><div>+ Mat Gmat = *a_Gmat, tGmat, matTrans;</div>
<div> MPI_Comm wcomm = ((PetscObject)Gmat)->comm;</div><div> const PetscScalar *vals;</div><div> const PetscInt *idx;</div><div>@@ -127,6 +127,10 @@</div><div> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr);</div>
<div> ierr = VecDestroy( &diag ); CHKERRQ(ierr);</div><div> </div><div>+ if( symm ) {</div><div>+ ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr);</div><div>+ }</div><div>
+</div><div> /* filter - dup zeros out matrix */</div><div> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr);</div><div> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr);</div>
<div>@@ -135,6 +139,12 @@</div><div> d_nnz[jj] = ncols;</div><div> o_nnz[jj] = ncols;</div><div> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div><div>+ if( symm ) {</div>
<div>+ ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div><div>+ d_nnz[jj] += ncols;</div><div>+ o_nnz[jj] += ncols;</div><div>+ ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div>
<div>+ }</div><div> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc;</div><div> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc;</div><div> }</div><div>@@ -142,6 +152,9 @@</div><div> CHKERRQ(ierr);</div><div>
ierr = PetscFree( d_nnz ); CHKERRQ(ierr); </div><div> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); </div><div>+ if( symm ) {</div><div>+ ierr = MatDestroy( &matTrans ); CHKERRQ(ierr);</div><div>+ }</div><div><div class="h5">
<div> </div><div><br></div><div><br></div><div><br></div><div><div>On Mar 14, 2012, at 5:53 PM, John Mousel wrote:</div><br><blockquote type="cite">Mark,<br><br>No change. Can you give me the location that you patched so I can check to make sure it pulled?<br>
I don't see it on the petsc-dev change log.<br><br>John<br><br><div class="gmail_quote">On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">John, I've committed these changes, give a try.<br>
<span><font color="#888888"><br>
Mark<br>
</font></span><div><div><br>
On Mar 14, 2012, at 3:46 PM, Satish Balay wrote:<br>
<br>
> This is the usual merge [with uncommited changes] issue.<br>
><br>
> You could use 'hg shelf' extension to shelve your local changes and<br>
> then do a merge [as Sean would suggest] - or do the merge in a<br>
> separate/clean clone [I normally do this..]<br>
><br>
> i.e<br>
> cd ~/Codes<br>
> hg clone petsc-dev petsc-dev-merge<br>
> cd petsc-dev-merge<br>
> hg pull ssh://<a href="mailto:petsc@petsc.cs.iit.edu" target="_blank">petsc@petsc.cs.iit.edu</a>//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge..<br>
> hg merge<br>
> hg commit<br>
> hg push ssh://<a href="mailto:petsc@petsc.cs.iit.edu" target="_blank">petsc@petsc.cs.iit.edu</a>//hg/petsc/petsc-dev<br>
><br>
> [now update your petsc-dev to latest]<br>
> cd ~/Codes/petsc-dev<br>
> hg pull<br>
> hg update<br>
><br>
> Satish<br>
><br>
> On Wed, 14 Mar 2012, Mark F. Adams wrote:<br>
><br>
>> Great, that seems to work.<br>
>><br>
>> I did a 'hg commit tools.c'<br>
>><br>
>> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this:<br>
>><br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update<br>
>> abort: crosses branches (merge branches or use --clean to discard changes)<br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge<br>
>> abort: outstanding uncommitted changes (use 'hg status' to list changes)<br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status<br>
>> M include/petscmat.h<br>
>> M include/private/matimpl.h<br>
>> M src/ksp/pc/impls/gamg/agg.c<br>
>> M src/ksp/pc/impls/gamg/gamg.c<br>
>> M src/ksp/pc/impls/gamg/gamg.h<br>
>> M src/ksp/pc/impls/gamg/geo.c<br>
>> M src/mat/coarsen/coarsen.c<br>
>> M src/mat/coarsen/impls/hem/hem.c<br>
>> M src/mat/coarsen/impls/mis/mis.c<br>
>><br>
>> Am I ready to do a push?<br>
>><br>
>> Thanks,<br>
>> Mark<br>
>><br>
>> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote:<br>
>><br>
>>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit.<br>
>>><br>
>>> Satish<br>
>>><br>
>>> On Wed, 14 Mar 2012, Mark F. Adams wrote:<br>
>>><br>
>>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this.<br>
>>>><br>
>>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file.<br>
>>>><br>
>>>> Anyone know how to delete a commit?<br>
>>>><br>
>>>> I've tried:<br>
>>>><br>
>>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f<br>
>>>> hg: unknown command 'strip'<br>
>>>> 'strip' is provided by the following extension:<br>
>>>><br>
>>>> mq manage a stack of patches<br>
>>>><br>
>>>> use "hg help extensions" for information on enabling extensions<br>
>>>><br>
>>>> But have not figured out how to load extensions.<br>
>>>><br>
>>>> Mark<br>
>>>><br>
>>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote:<br>
>>>><br>
>>>>> Mark,<br>
>>>>><br>
>>>>> I have a non-symmetric matrix. I am running with the following options.<br>
>>>>><br>
>>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual<br>
>>>>><br>
>>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error:<br>
>>>>><br>
>>>>><br>
>>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>>>> [0]PETSC ERROR: Argument out of range!<br>
>>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc!<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500<br>
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named <a href="http://wv.iihr.uiowa.edu/" target="_blank">wv.iihr.uiowa.edu</a> by jmousel Wed Mar 14 11:51:35 2012<br>
>>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib<br>
>>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012<br>
>>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
>>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c<br>
>>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c<br>
>>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c<br>
>>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c<br>
>>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c<br>
>>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c<br>
>>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c<br>
>>>>><br>
>>>>><br>
>>>>> John<br>
>>>>><br>
>>>>><br>
>>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> wrote:<br>
>>>>><br>
>>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote:<br>
>>>>><br>
>>>>>> Mark,<br>
>>>>>><br>
>>>>>> The matrix is asymmetric. Does this require the setting of an option?<br>
>>>>><br>
>>>>> Yes: -pc_gamg_sym_graph<br>
>>>>><br>
>>>>> Mark<br>
>>>>><br>
>>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code.<br>
>>>>>><br>
>>>>>> John<br>
>>>>>><br>
>>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> wrote:<br>
>>>>>><br>
>>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote:<br>
>>>>>><br>
>>>>>>> I'm getting the following error when using GAMG.<br>
>>>>>>><br>
>>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed.<br>
>>>>>><br>
>>>>>> Is it possible that your matrix is structurally asymmetric?<br>
>>>>>><br>
>>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert).<br>
>>>>>><br>
>>>>>>><br>
>>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting<br>
>>>>>>><br>
>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type:<br>
>>>>>>> see <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" target="_blank">http://www.mcs.anl.gov/petsc/documentation/installation.html#external</a>!<br>
>>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given!<br>
>>>>>>><br>
>>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning.<br>
>>>>>>><br>
>>>>>><br>
>>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now.<br>
>>>>>><br>
>>>>>> Mark<br>
>>>>>><br>
>>>>>>> John<br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>><br>
>><br>
><br>
><br>
<br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></blockquote></div><br>