<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">You also want: -pc_gamg_agg_nsmooths 1<div><br></div><div>You are running plain aggregation. If it is Poisson then smoothing is good.</div><div><br></div><div>Is this problem singular? Can you try running ML with these parameters and see if its performance degrades? The ML implementation uses the PETSC infrastructure and uses a very similar algorithm to GAMG-SA. We should be able to get these two to match pretty well.</div><div><br></div><div>Mark<br><div><br></div><div><br><div><div>On Mar 15, 2012, at 12:21 PM, John Mousel wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Mark,<br><br>I ran with those options removed (see the run options listed below). Things actually got slightly worse. Now it's up to 142 iterations. I have attached the ksp_view output.<br><br>-ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -mg_levels_ksp_type richardson -mg_levels_pc_type sor -pc_gamg_verbose 1 <br>
<br><br>John<br><br><br><div class="gmail_quote">On Thu, Mar 15, 2012 at 10:55 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">John, can you run again with: -pc_gamg_verbose 1<div><br></div><div>And I would not use: -pc_mg_levels 4 -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8</div><div>
<br></div><div>1) I think -mg_coarse_ksp_type preonly and -mg_coarse_pc_sor_its 8 do not do what you think. I think this is the same as 1 iteration. I think you want 'richardson' not 'preonly'.</div><div>
<br></div><div>2) Why are you using sor as the coarse solver? If your problem is singular then you want to use as many levels as possible to get the coarse grid to be tiny. I'm pretty sure HYPRE ignores the coarse solver parameters. But ML uses them and it is converging well.</div>
<div><br></div><div>3) I would not specify the number of levels. GAMG, and I think the rest, have internal logic for stopping a the right level. If the coarse level is large and you use just 8 iterations of sor then convergence will suffer.</div>
<div><br></div><div>Mark</div><div><br><div><div><div class="h5"><div>On Mar 15, 2012, at 11:13 AM, John Mousel wrote:</div><br></div></div><blockquote type="cite"><div><div class="h5">Mark,<br><br>The changes pulled through this morning. I've run it with the options<br>
<br>-ksp_type bcgsl -pc_type gamg -pc_gamg_sym_graph -ksp_diagonal_scale -ksp_diagonal_scale_fix -pc_mg_levels 4 -mg_levels_ksp_type richardson -mg_levels_pc_type sor -mg_coarse_ksp_type preonly -mg_coarse_pc_type sor -mg_coarse_pc_sor_its 8<br>
<br>and it converges in the true residual, but it's not converging as fast as anticpated. The matrix arises from a non-symmetric discretization of the Poisson equation. The solve takes GAMG 114 iterations, whereas ML takes 24 iterations, BoomerAMG takes 22 iterations, and -ksp_type bcgsl -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 4 takes around 170. I've attached the -ksp_view results for ML,GAMG, and HYPRE. I've attempted to make all the options the same on all levels for ML and GAMG. <br>
<br>Any thoughts?<br><br>John<br><br><br><div class="gmail_quote">On Wed, Mar 14, 2012 at 6:04 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Humm, I see it with hg view (appended). </div><div><br></div><div>Satish, my main repo looks hosed. I see this:</div>
<div><br></div><div><div>~/Codes/petsc-dev>hg update</div><div><div>abort: crosses branches (merge branches or use --clean to discard changes)</div></div><div>~/Codes/petsc-dev>hg merge</div><div>abort: branch 'default' has 3 heads - please merge with an explicit rev</div>
<div>(run 'hg heads .' to see heads)</div><div>~/Codes/petsc-dev>hg heads</div><div>changeset: 22496:8e2a98268179</div><div>tag: tip</div><div>user: Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></div>
<div>date: Wed Mar 14 16:42:25 2012 -0500</div><div>files: src/vec/is/interface/f90-custom/zindexf90.c src/vec/vec/interface/f90-custom/zvectorf90.c</div><div>description:</div><div>undoing manually changes I put in because Satish had a better fix</div>
<div><br></div><div><br></div><div>changeset: 22492:bda4df63072d</div><div>user: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></div><div>date: Wed Mar 14 17:39:52 2012 -0400</div>
<div>files: src/ksp/pc/impls/gamg/tools.c</div><div>description:</div><div>fix for unsymmetric matrices.</div><div><br></div><div><br></div><div>changeset: 22469:b063baf366e4</div><div>user: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></div>
<div>date: Wed Mar 14 14:22:28 2012 -0400</div><div>files: src/ksp/pc/impls/gamg/tools.c</div><div>description:</div><div>added fix for preallocation for unsymetric matrices.</div><div><br></div></div><div>Mark</div>
<div><br></div><div>my 'hg view' on my merge repo:</div><div><br></div><div>Revision: 22492</div><div>Branch: default</div><div>Author: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> 2012-03-14 17:39:52</div>
<div>Committer: Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> 2012-03-14 17:39:52</div><div>Tags: tip</div><div>Parent: 22491:451bbbd291c2 (Small fixes to the BT linesearch)</div>
<div><br></div><div> fix for unsymmetric matrices.</div><div><br></div><div><br></div><div>------------------------ src/ksp/pc/impls/gamg/tools.c ------------------------</div><div>@@ -103,7 +103,7 @@</div><div> PetscErrorCode ierr;</div>
<div> PetscInt Istart,Iend,Ii,jj,ncols,nnz0,nnz1, NN, MM, nloc;</div><div> PetscMPIInt mype, npe;</div><div>- Mat Gmat = *a_Gmat, tGmat;</div><div>+ Mat Gmat = *a_Gmat, tGmat, matTrans;</div>
<div> MPI_Comm wcomm = ((PetscObject)Gmat)->comm;</div><div> const PetscScalar *vals;</div><div> const PetscInt *idx;</div><div>@@ -127,6 +127,10 @@</div><div> ierr = MatDiagonalScale( Gmat, diag, diag ); CHKERRQ(ierr);</div>
<div> ierr = VecDestroy( &diag ); CHKERRQ(ierr);</div><div> </div><div>+ if( symm ) {</div><div>+ ierr = MatTranspose( Gmat, MAT_INITIAL_MATRIX, &matTrans ); CHKERRQ(ierr);</div><div>+ }</div>
<div>
+</div><div> /* filter - dup zeros out matrix */</div><div> ierr = PetscMalloc( nloc*sizeof(PetscInt), &d_nnz ); CHKERRQ(ierr);</div><div> ierr = PetscMalloc( nloc*sizeof(PetscInt), &o_nnz ); CHKERRQ(ierr);</div>
<div>@@ -135,6 +139,12 @@</div><div> d_nnz[jj] = ncols;</div><div> o_nnz[jj] = ncols;</div><div> ierr = MatRestoreRow(Gmat,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div><div>+ if( symm ) {</div>
<div>+ ierr = MatGetRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div><div>+ d_nnz[jj] += ncols;</div><div>+ o_nnz[jj] += ncols;</div><div>+ ierr = MatRestoreRow(matTrans,Ii,&ncols,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr);</div>
<div>+ }</div><div> if( d_nnz[jj] > nloc ) d_nnz[jj] = nloc;</div><div> if( o_nnz[jj] > (MM-nloc) ) o_nnz[jj] = MM - nloc;</div><div> }</div><div>@@ -142,6 +152,9 @@</div><div> CHKERRQ(ierr);</div><div>
ierr = PetscFree( d_nnz ); CHKERRQ(ierr); </div><div> ierr = PetscFree( o_nnz ); CHKERRQ(ierr); </div><div>+ if( symm ) {</div><div>+ ierr = MatDestroy( &matTrans ); CHKERRQ(ierr);</div><div>+ }</div><div>
<div>
<div> </div><div><br></div><div><br></div><div><br></div><div><div>On Mar 14, 2012, at 5:53 PM, John Mousel wrote:</div><br><blockquote type="cite">Mark,<br><br>No change. Can you give me the location that you patched so I can check to make sure it pulled?<br>
I don't see it on the petsc-dev change log.<br><br>John<br><br><div class="gmail_quote">On Wed, Mar 14, 2012 at 4:40 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">John, I've committed these changes, give a try.<br>
<span><font color="#888888"><br>
Mark<br>
</font></span><div><div><br>
On Mar 14, 2012, at 3:46 PM, Satish Balay wrote:<br>
<br>
> This is the usual merge [with uncommited changes] issue.<br>
><br>
> You could use 'hg shelf' extension to shelve your local changes and<br>
> then do a merge [as Sean would suggest] - or do the merge in a<br>
> separate/clean clone [I normally do this..]<br>
><br>
> i.e<br>
> cd ~/Codes<br>
> hg clone petsc-dev petsc-dev-merge<br>
> cd petsc-dev-merge<br>
> hg pull ssh://<a href="mailto:petsc@petsc.cs.iit.edu" target="_blank">petsc@petsc.cs.iit.edu</a>//hg/petsc/petsc-dev #just to be sure, look for latest chagnes before merge..<br>
> hg merge<br>
> hg commit<br>
> hg push ssh://<a href="mailto:petsc@petsc.cs.iit.edu" target="_blank">petsc@petsc.cs.iit.edu</a>//hg/petsc/petsc-dev<br>
><br>
> [now update your petsc-dev to latest]<br>
> cd ~/Codes/petsc-dev<br>
> hg pull<br>
> hg update<br>
><br>
> Satish<br>
><br>
> On Wed, 14 Mar 2012, Mark F. Adams wrote:<br>
><br>
>> Great, that seems to work.<br>
>><br>
>> I did a 'hg commit tools.c'<br>
>><br>
>> and I want to push this file only. I guess its the only thing in the change set so 'hg push' should be fine. But I see this:<br>
>><br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg update<br>
>> abort: crosses branches (merge branches or use --clean to discard changes)<br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg merge<br>
>> abort: outstanding uncommitted changes (use 'hg status' to list changes)<br>
>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg status<br>
>> M include/petscmat.h<br>
>> M include/private/matimpl.h<br>
>> M src/ksp/pc/impls/gamg/agg.c<br>
>> M src/ksp/pc/impls/gamg/gamg.c<br>
>> M src/ksp/pc/impls/gamg/gamg.h<br>
>> M src/ksp/pc/impls/gamg/geo.c<br>
>> M src/mat/coarsen/coarsen.c<br>
>> M src/mat/coarsen/impls/hem/hem.c<br>
>> M src/mat/coarsen/impls/mis/mis.c<br>
>><br>
>> Am I ready to do a push?<br>
>><br>
>> Thanks,<br>
>> Mark<br>
>><br>
>> On Mar 14, 2012, at 2:44 PM, Satish Balay wrote:<br>
>><br>
>>> If commit is the last hg operation that you've done - then 'hg rollback' would undo this commit.<br>
>>><br>
>>> Satish<br>
>>><br>
>>> On Wed, 14 Mar 2012, Mark F. Adams wrote:<br>
>>><br>
>>>> Damn, I'm not preallocating the graph perfectly for unsymmetric matrices and PETSc now dies on this.<br>
>>>><br>
>>>> I have a fix but I committed it with other changes that I do not want to commit. The changes are all in one file so I should be able to just commit this file.<br>
>>>><br>
>>>> Anyone know how to delete a commit?<br>
>>>><br>
>>>> I've tried:<br>
>>>><br>
>>>> ~/Codes/petsc-dev/src/ksp/pc/impls/gamg>hg strip 22487:26ffb9eef17f<br>
>>>> hg: unknown command 'strip'<br>
>>>> 'strip' is provided by the following extension:<br>
>>>><br>
>>>> mq manage a stack of patches<br>
>>>><br>
>>>> use "hg help extensions" for information on enabling extensions<br>
>>>><br>
>>>> But have not figured out how to load extensions.<br>
>>>><br>
>>>> Mark<br>
>>>><br>
>>>> On Mar 14, 2012, at 12:54 PM, John Mousel wrote:<br>
>>>><br>
>>>>> Mark,<br>
>>>>><br>
>>>>> I have a non-symmetric matrix. I am running with the following options.<br>
>>>>><br>
>>>>> -pc_type gamg -pc_gamg_sym_graph -ksp_monitor_true_residual<br>
>>>>><br>
>>>>> and with the inclusion of -pc_gamg_sym_graph, I get a new malloc error:<br>
>>>>><br>
>>>>><br>
>>>>> 0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>>>> [0]PETSC ERROR: Argument out of range!<br>
>>>>> [0]PETSC ERROR: New nonzero at (5150,9319) caused a malloc!<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: Petsc Development HG revision: 587b25035091aaa309c87c90ac64c13408ecf34e HG Date: Wed Mar 14 09:22:54 2012 -0500<br>
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: ../JohnRepo/VFOLD_exe on a linux-deb named <a href="http://wv.iihr.uiowa.edu/" target="_blank">wv.iihr.uiowa.edu</a> by jmousel Wed Mar 14 11:51:35 2012<br>
>>>>> [0]PETSC ERROR: Libraries linked from /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/linux-debug/lib<br>
>>>>> [0]PETSC ERROR: Configure run at Wed Mar 14 09:46:39 2012<br>
>>>>> [0]PETSC ERROR: Configure options --download-blacs=1 --download-hypre=1 --download-metis=1 --download-ml=1 --download-mpich=1 --download-parmetis=1 --download-scalapack=1 --with-blas-lapack-dir=/opt/intel11/mkl/lib/em64t --with-cc=gcc --with-cmake=/usr/local/bin/cmake --with-cxx=g++ --with-fc=ifort PETSC_ARCH=linux-debug<br>
>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
>>>>> [0]PETSC ERROR: MatSetValues_MPIAIJ() line 506 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c<br>
>>>>> [0]PETSC ERROR: MatSetValues() line 1141 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/mat/interface/matrix.c<br>
>>>>> [0]PETSC ERROR: scaleFilterGraph() line 155 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/tools.c<br>
>>>>> [0]PETSC ERROR: PCGAMGgraph_AGG() line 865 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/agg.c<br>
>>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 516 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/impls/gamg/gamg.c<br>
>>>>> [0]PETSC ERROR: PCSetUp() line 832 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/pc/interface/precon.c<br>
>>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c<br>
>>>>> [0]PETSC ERROR: KSPSolve() line 385 in /home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/src/ksp/ksp/interface/itfunc.c<br>
>>>>><br>
>>>>><br>
>>>>> John<br>
>>>>><br>
>>>>><br>
>>>>> On Wed, Mar 14, 2012 at 11:27 AM, Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> wrote:<br>
>>>>><br>
>>>>> On Mar 14, 2012, at 11:56 AM, John Mousel wrote:<br>
>>>>><br>
>>>>>> Mark,<br>
>>>>>><br>
>>>>>> The matrix is asymmetric. Does this require the setting of an option?<br>
>>>>><br>
>>>>> Yes: -pc_gamg_sym_graph<br>
>>>>><br>
>>>>> Mark<br>
>>>>><br>
>>>>>> I pulled petsc-dev this morning, so I should have (at least close to) the latest code.<br>
>>>>>><br>
>>>>>> John<br>
>>>>>><br>
>>>>>> On Wed, Mar 14, 2012 at 10:54 AM, Mark F. Adams <<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>> wrote:<br>
>>>>>><br>
>>>>>> On Mar 14, 2012, at 11:08 AM, John Mousel wrote:<br>
>>>>>><br>
>>>>>>> I'm getting the following error when using GAMG.<br>
>>>>>>><br>
>>>>>>> petsc-dev/src/ksp/pc/impls/gamg/agg.c:508: smoothAggs: Assertion `sgid==-1' failed.<br>
>>>>>><br>
>>>>>> Is it possible that your matrix is structurally asymmetric?<br>
>>>>>><br>
>>>>>> This code is evolving fast and so you will need to move to the dev version if you are not already using it. (I think I fixed a bug that hit this assert).<br>
>>>>>><br>
>>>>>>><br>
>>>>>>> When I try to alter the type of aggregation at the command line using -pc_gamg_type pa, I'm getting<br>
>>>>>>><br>
>>>>>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>>>>>> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing external package needed for type:<br>
>>>>>>> see <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" target="_blank">http://www.mcs.anl.gov/petsc/documentation/installation.html#external</a>!<br>
>>>>>>> [1]PETSC ERROR: Unknown GAMG type pa given!<br>
>>>>>>><br>
>>>>>>> Has there been a change in the aggregation options? I just pulled petsc-dev this morning.<br>
>>>>>>><br>
>>>>>><br>
>>>>>> Yes, this option is gone now. You can use -pc_gamg_type agg for now.<br>
>>>>>><br>
>>>>>> Mark<br>
>>>>>><br>
>>>>>>> John<br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>><br>
>><br>
><br>
><br>
<br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></blockquote></div><br>
</div></div><span><GAMG_kspview.txt></span><span><ML_kspview.txt></span><span><HYPRE_kspview.txt></span></blockquote></div><br></div></div></blockquote></div><br>
<span><GAMG_kspview.txt></span></blockquote></div><br></div></div></body></html>