Mark,<br><br>The problem is indeed non-symmetric. We went back and forth in March about this problem. I think we ended up concluding that the coarse size couldn't get too small or the null-space presented problems. When I did get it to work, I tried to scale it up, and on my local university cluster, it seemed to just hang when the core counts got above something like 16 cores. I don't really trust that machine though. It's new and has been plagued by hardware incompatability issues since day 1. I could re-examine this on Kraken. Also, what option are you talking about with ML. I thought I had tried all the -pc_ml_CoarsenScheme options, but I could be wrong.<br>
<br>John<br><br> <br><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 11:30 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">What problems are you having again with GAMG? Are you problems unsymmetric?<div><br>
</div><div>ML has several coarsening strategies available and I think the default does aggregation locally and does not aggregate across processor subdomains. If you have poorly shaped domains then you want to use a global coarsening method (these are not expensive).</div>
<span class="HOEnZb"><font color="#888888"><div><br></div><div>Mark</div></font></span><div><div class="h5"><div><br><div><div>On Jul 9, 2012, at 12:17 PM, John Mousel wrote:</div><br><blockquote type="cite">Mark,<br><br>
I still haven't had much luck getting GAMG to work consistently for my Poisson problem. ML seems to work nicely on low core counts, but I have a problem where I can get long thin portions of grid on some processors instead of nice block like chunks at high core counts, which leads to a pretty tough time for ML. <br>
<br>John<br><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 10:58 AM, John Mousel <span dir="ltr"><<a href="mailto:john.mousel@gmail.com" target="_blank">john.mousel@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Getting rid of the Hypre option seemed to be the trick. <br><div><div><br><div class="gmail_quote">On Mon, Jul 9, 2012 at 10:40 AM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu" target="_blank">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Google PTL_NO_SPACE and you will find some NERSC presentations on how to go about fixing this. (I have run into these problems years ago but forget the issues)<br>
<br>
Also, I would try running with a Jacobi solver to see if that fixes the problem. If so then you might try<br>
<br>
-pc_type gamg<br>
-pc_gamg_agg_nsmooths 1<br>
-pc_gamg_type agg<br>
<br>
This is a built in AMG solver so perhaps it plays nicer with resources ...<br>
<span><font color="#888888"><br>
Mark<br>
</font></span><div><div><br>
On Jul 9, 2012, at 10:57 AM, John Mousel wrote:<br>
<br>
> I'm running on Kraken and am currently working with 4320 cores. I get the following error in KSPSolve.<br>
><br>
> [2711]: (/ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046) PtlMEInsert failed with error : PTL_NO_SPACE<br>
> MHV_exe: /ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046: MPIDI_CRAY_ptldev_desc_pkt: Assertion `0' failed.<br>
> forrtl: error (76): Abort trap signal<br>
> Image PC Routine Line Source<br>
> MHV_exe 00000000014758CB Unknown Unknown Unknown<br>
> MHV_exe 000000000182ED43 Unknown Unknown Unknown<br>
> MHV_exe 0000000001829460 Unknown Unknown Unknown<br>
> MHV_exe 00000000017EDE3E Unknown Unknown Unknown<br>
> MHV_exe 00000000017B3FE6 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B3738 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B2B12 Unknown Unknown Unknown<br>
> MHV_exe 00000000017B428F Unknown Unknown Unknown<br>
> MHV_exe 000000000177FCE1 Unknown Unknown Unknown<br>
> MHV_exe 0000000001590A43 Unknown Unknown Unknown<br>
> MHV_exe 00000000014F909B Unknown Unknown Unknown<br>
> MHV_exe 00000000014FF53B Unknown Unknown Unknown<br>
> MHV_exe 00000000014A4E25 Unknown Unknown Unknown<br>
> MHV_exe 0000000001487D57 Unknown Unknown Unknown<br>
> MHV_exe 000000000147F726 Unknown Unknown Unknown<br>
> MHV_exe 000000000137A8D3 Unknown Unknown Unknown<br>
> MHV_exe 0000000000E97BF2 Unknown Unknown Unknown<br>
> MHV_exe 000000000098EAF1 Unknown Unknown Unknown<br>
> MHV_exe 0000000000989C20 Unknown Unknown Unknown<br>
> MHV_exe 000000000097A9C2 Unknown Unknown Unknown<br>
> MHV_exe 000000000082FF2D axbsolve_ 539 PetscObjectsOperations.F90<br>
><br>
> This is somewhere in KSPSolve. Is there an MPICH environment variable that needs tweaking? I couldn't really find much on this particular error.<br>
> The solver is BiCGStab with Hypre as a preconditioner.<br>
><br>
> -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor<br>
><br>
> Thanks,<br>
><br>
> John<br>
<br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br>