<div dir="ltr">Barry, I have never seen an empty matrix for this problem. I am in my branch (mark/fix-gamg-asm-aggs), which basically just has the fix to add a dummy block for BCs to make ASM work properly, added to your branch.<div><br></div><div>This (appended) is what I get with 1 proc (it is fine, it is solving) . With 8 procs I get this stack trace. I will keep digging.</div><div><br></div><div><div><div><br></div><div>#8 main (argc=36, args=0x7fffffff4628) at /global/homes/m/madams/petsc/src/ksp/ksp/examples/tutorials/ex56.c:280 (at 0x0000000000409f2e)</div><div>#7 KSPSetUp (ksp=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:393 (at 0x000000000075d99f)</div><div>#6 PCSetUp (pc=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:968 (at 0x00000000007531da)</div><div>#5 PCSetUp_GAMG (pc=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:672 (at 0x0000000000c28a81)</div><div>#4 PCSetUp_MG (pc=0x4760330) at /global/u2/m/madams/petsc/src/ksp/pc/impls/mg/mg.c:781 (at 0x0000000000c87f7e)</div><div>#3 KSPSetUp (ksp=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:393 (at 0x000000000075d99f)</div><div>#2 PCSetUp (pc=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:968 (at 0x00000000007531da)</div><div>#1 PCSetUp_ASM (pc=0x46e2f20) at /global/u2/m/madams/petsc/src/ksp/pc/impls/asm/asm.c:383 (at 0x0000000000c5c499)</div><div>#0 MatGetSubMatrices (mat=0x46e2f20, n=63418000, irow=0x21, icol=0x4a8, scall=MAT_INITIAL_MATRIX, submat=0x2) at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:6780 (at 0x00000000004e8bcd)</div><div><br></div></div><div><br></div><div><br></div><div><div>06:32 134 nid00015 ~/petsc/src/ksp/ksp/examples/tutorials$ srun -n 1 ./ex56 -ne 3 -alpha 1.e-3 -ksp_type cg -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -two_solves true -ksp_converged_reason -use_mat_nearnullspace -mg_levels_esteig_ksp_type cg -pc_gamg_square_graph 1 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -gamg_est_ksp_type cg -gamg_est_ksp_max_it 20 -pc_gamg_asm_use_agg -mg_levels_sub_pc_type lu -ksp_monitor </div><div> 0 KSP Residual norm 2.553528059796e+02 </div><div> 1 KSP Residual norm 1.285760737388e+01 </div><div> 2 KSP Residual norm 5.896306650166e-01 </div><div> 3 KSP Residual norm 3.206061310861e-02 </div><div> 4 KSP Residual norm 3.287356003543e-03 </div><div> 5 KSP Residual norm 1.695272457611e-04 </div><div>Linear solve converged due to CONVERGED_RTOL iterations 5 </div><div> 0 KSP Residual norm 2.553528059796e-03 </div><div> 1 KSP Residual norm 1.285760737388e-04 </div><div> 2 KSP Residual norm 5.896306650166e-06 </div><div> 3 KSP Residual norm 3.206061310861e-07 </div><div> 4 KSP Residual norm 3.287356003543e-08 </div><div> 5 KSP Residual norm 1.695272457611e-09 </div><div>Linear solve converged due to CONVERGED_RTOL iterations 5 </div><div> 0 KSP Residual norm 2.553528059796e-08 </div><div> 1 KSP Residual norm 1.285760737388e-09 </div><div> 2 KSP Residual norm 5.896306650166e-11 </div><div> 3 KSP Residual norm 3.206061310861e-12 </div><div> 4 KSP Residual norm 3.287356003543e-13 </div><div> 5 KSP Residual norm 1.695272457611e-14 </div><div>Linear solve converged due to CONVERGED_RTOL iterations 5 </div><div>[0]main |b-Ax|/|b|=9.327411e-06, |b|=7.453560e+00, emax=9.907752e-01 </div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 29, 2016 at 1:05 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
Mark<br>
<br>
When I run with exactly the options in this file on either 1 or 8 process and MatView() the matrix you pass to KSP the matrix has ALL zeros. So of course the GAMG solver is not happy. Is the matrix suppose to have all zeros? When I run the command line options in the make file for runex56 the matrix has nonzero values in it and GAMG runs fine.<br>
<br>
Barry<br>
<div><div><br>
> On Jun 27, 2016, at 6:56 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
><br>
> Barry,<br>
><br>
> I am having problems with your branch on ksp/ex56. First I was getting an error with the increase of the overlap that you added (I don't want that to be the default). So I set this back to 0 in gamg.c.<br>
><br>
> Next, I get the same error with the vec out of range because ex56 has BCs in the matrix. I created a branch mark/fix-gamg-asm-aggs from your branch and added the dummy ASM block. But I am getting all kinds of errors. I get errors in get MatSubMatrix and a funny error when it tries to free a prefix string.<br>
><br>
> I've attached the valgrind output of one process, of 8. There is a lot of noise with string operations but it looks like the first non-string thing is in MatGetSubMatrices. ex56 is elasticity. Maybe there is a block size problem?<br>
><br>
><br>
</div></div>> <out_val_1><br>
<br>
</blockquote></div><br></div></div>