Mark,<br><br>I just tried the following options on Kraken on 1200 cores:<br><br>-pres_ksp_type bcgsl -pres_pc_type gamg -pres_pc_gamg_type agg -pres_pc_gamg_agg_nsmooths 1 -pres_pc_gamg_threshold 0.05 -pres_mg_levels_ksp_type richardson -pres_mg_levels_pc_type sor -pres_mg_coarse_ksp_typ<br>
e richardson -pres_mg_coarse_pc_type sor -pres_mg_coarse_pc_sor_its 4<br><br>It hung at <br><br>[0]PCSetData_AGG bs=1 MM=10672<br><br>for nearly 15 minutes. I take it this is not normal.<br><br>John<br><br><br><br><div class="gmail_quote">
On Mon, Jul 9, 2012 at 2:41 PM, John Mousel <span dir="ltr"><<a href="mailto:john.mousel@gmail.com" target="_blank">john.mousel@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can you clarify what you mean by null-space cleaning. I just run SOR on the coarse grid.<div class="HOEnZb"><div class="h5"><br><br><br><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 11:52 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div><div>On Jul 9, 2012, at 12:39 PM, John Mousel wrote:</div><br>
</div><blockquote type="cite">Mark,<br><br><div>The problem is indeed non-symmetric. We went back and forth in March about this problem. I think we ended up concluding that the coarse size couldn't get too small or the null-space presented problems. </div>
</blockquote><div><br></div><div>Oh its singular. I forget what the issues were but an iterative coarse grid solver should be fine for singular problems, perhaps with null space cleaning if the kernel is sneaking in. Actually there is an SVD coarse grid solver:</div>
<div><br></div><div><font face="courier new, monospace">-mg_coars</font><span style="font-family:'courier new',monospace">e_pc_type svd</span></div><div><br></div><div>That is the most robust.</div><div>
<br><blockquote type="cite">When I did get it to work, I tried to scale it up, and on my local university cluster, it seemed to just hang when the core counts got above something like 16 cores. I don't really trust that machine though. </blockquote>
<div><br></div></div><div>That's the machine. GAMG does have some issues but I've not seen it hang.</div><div><br><blockquote type="cite">It's new and has been plagued by hardware incompatability issues since day 1. I could re-examine this on Kraken. Also, what option are you talking about with ML. I thought I had tried all the -pc_ml_CoarsenScheme options, but I could be wrong.<br>
</blockquote><div><br></div></div><div>This sounds like the right one. I try to be careful in my solvers to be invariant to subdomain shapes and sizes and I think Ray Tuminaro (ML developer) at least has options that should be careful about this also. But I don't know much about what they are deploying these days.</div>
<span><font color="#888888"><div><br></div><div>Mark</div></font></span><div><div><br><blockquote type="cite">
<br>John<br><br> <br><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 11:30 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">What problems are you having again with GAMG? Are you problems unsymmetric?<div><br>
</div><div>ML has several coarsening strategies available and I think the default does aggregation locally and does not aggregate across processor subdomains. If you have poorly shaped domains then you want to use a global coarsening method (these are not expensive).</div>
<span><font color="#888888"><div><br></div><div>Mark</div></font></span><div><div><div><br><div><div>On Jul 9, 2012, at 12:17 PM, John Mousel wrote:</div><br><blockquote type="cite">Mark,<br><br>
I still haven't had much luck getting GAMG to work consistently for my Poisson problem. ML seems to work nicely on low core counts, but I have a problem where I can get long thin portions of grid on some processors instead of nice block like chunks at high core counts, which leads to a pretty tough time for ML. <br>
<br>John<br><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 10:58 AM, John Mousel <span dir="ltr"><<a href="mailto:john.mousel@gmail.com" target="_blank">john.mousel@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Getting rid of the Hypre option seemed to be the trick. <br><div><div><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 10:40 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Google PTL_NO_SPACE and you will find some NERSC presentations on how to go about fixing this. (I have run into these problems years ago but forget the issues)<br>
<br>
Also, I would try running with a Jacobi solver to see if that fixes the problem. If so then you might try<br>
<br>
-pc_type gamg<br>
-pc_gamg_agg_nsmooths 1<br>
-pc_gamg_type agg<br>
<br>
This is a built in AMG solver so perhaps it plays nicer with resources ...<br>
<span><font color="#888888"><br>
Mark<br>
</font></span><div><div><br>
On Jul 9, 2012, at 10:57 AM, John Mousel wrote:<br>
<br>
> I'm running on Kraken and am currently working with 4320 cores. I get the following error in KSPSolve.<br>
><br>
> [2711]: (/ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046) PtlMEInsert failed with error : PTL_NO_SPACE<br>
> MHV_exe: /ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046: MPIDI_CRAY_ptldev_desc_pkt: Assertion `0' failed.<br>
> forrtl: error (76): Abort trap signal<br>
> Image PC Routine Line Source<br>
> MHV_exe 00000000014758CB Unknown Unknown Unknown<br>
> MHV_exe 000000000182ED43 Unknown Unknown Unknown<br>
> MHV_exe 0000000001829460 Unknown Unknown Unknown<br>
> MHV_exe 00000000017EDE3E Unknown Unknown Unknown<br>
> MHV_exe 00000000017B3FE6 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B3738 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B2B12 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B428F Unknown Unknown Unknown<br>
> MHV_exe 000000000177FCE1 Unknown Unknown Unknown<br>
> MHV_exe 0000000001590A43 Unknown Unknown Unknown<br>
> MHV_exe 00000000014F909B Unknown Unknown Unknown<br>
> MHV_exe 00000000014FF53B Unknown Unknown Unknown<br>
> MHV_exe 00000000014A4E25 Unknown Unknown Unknown<br>
> MHV_exe 0000000001487D57 Unknown Unknown Unknown<br>
> MHV_exe 000000000147F726 Unknown Unknown Unknown<br>
> MHV_exe 000000000137A8D3 Unknown Unknown Unknown<br>
> MHV_exe 0000000000E97BF2 Unknown Unknown Unknown<br>
> MHV_exe 000000000098EAF1 Unknown Unknown Unknown<br>
> MHV_exe 0000000000989C20 Unknown Unknown Unknown<br>
> MHV_exe 000000000097A9C2 Unknown Unknown Unknown<br>
> MHV_exe 000000000082FF2D axbsolve_ 539 PetscObjectsOperations.F90<br>
><br>
> This is somewhere in KSPSolve. Is there an MPICH environment variable that needs tweaking? I couldn't really find much on this particular error.<br>
> The solver is BiCGStab with Hypre as a preconditioner.<br>
><br>
> -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor<br>
><br>
> Thanks,<br>
><br>
> John<br>
<br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br>
</blockquote></div></div></div><br></div></blockquote></div><br>
</div></div></blockquote></div><br>