<div dir="ltr">The null space for GAMG is not critical but useful for elasticity. If you in fact have an indefinite operator (eg, not "pinned) the you need to use an iterative coarse grid solver. You are using '-fieldsplit_0_pc_type gamg -fieldsplit_0_ksp_type preonly'. And you have a hard elasticity problem. You are going to want to start with a stronger solver. Use cg instead of preonly. As Matt said start with MUMPS, then go to CG/GAMG, then you can see how far you can cut the _0_ solver down.<div>Mark</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 23, 2014 at 11:51 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Thu, Oct 23, 2014 at 10:48 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div text="#000000" bgcolor="#ffffff">
Matt<br>
<br>
On 10/23/2014 09:54 AM, Matthew Knepley wrote:
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Oct 23, 2014 at 9:27 AM,
Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
Sorry about that (I always forget it). The output for
the smallest problem is now attached (see log.txt). I
am also attaching some results that compare results
obtained using FS/LSC and the direct solver (MUMPS),
again for the smallest problem. The difference, as you
can see is insignificant O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do you use MUMPS if you have a saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used -pc_type lu -pc_factor_mat_solver_package mumps.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>2) You can see from the output that something is
seriously wrong with the preconditioner. It looks like it
has a null space.</div>
<div> Did you add the elastic null modes to GAMG? Without
this, it is not going to work. We have helper functions
for this:</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could just copy that code. And then use</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't see it in the output, so I think this is your
problem.</div>
<div><br>
</div>
<div>In order to test, I would first use MUMPS as the A00
solver and get the Schur stuff worked out. Then I would</div>
<div>replace MUMPS with GAMG and tune it until I get back my
original convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try this with MatNullSpaceCreateRigidBody. Btw does it matter
if some nodes are pinned?<br></div></blockquote><div><br></div></span><div>No these are null modes of the operator, not of the particular problem.</div><div><br></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#ffffff">
Tabrez<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> Also, I did pass 'upper' and 'full' to
'-pc_fieldsplit_schur_factorization_type' but the
iteration count doesn't improve (in fact, it increases
slightly). The attached log is with 'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014 07:46 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Oct 23, 2014 at
7:20 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hello<br>
<br>
I am using the following options (below) for
solving linear elasticity/poroelasticity
problems involving slip between two surfaces
involving non-trivial geometries, i.e.,
elements with high aspect ratios, large
contrasts in material properties etc. The
constraints are imposed using Lagrange
Multipliers. <br>
<br>
A picture (shows displacement magnitude) is
attached. The boundary nodes, i.e., the base
and the four side are pinned.<br>
<br>
The following options appear to work well
for the saddle point problem:<br>
<br>
<tt>-pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type gamg
-fieldsplit_0_ksp_type preonly
-fieldsplit_1_pc_type lsc
-fieldsplit_1_ksp_type preonly
-pc_fieldsplit_schur_fact_type lower
-ksp_monitor</tt><br>
<br>
However, the number of iterations keep on
increasing with the problems size (see
attached plot), e.g.,<br>
<br>
<tt>120K Tets <b>507</b> Iterations
(KSP Residual norm 8.827362494659e-05)</tt><tt>
in 17 secs on 3 cores<br>
</tt><tt>1 Million Tets <b>1374</b>
Iterations (KSP Residual norm
7.164704416296e-05)</tt><tt> in 117 secs
on 20 cores<br>
</tt><tt>8 Million Tets <b>2495</b>
Iterations (KSP Residual norm
9.101247550026e-05) in 225 secs on 160
cores</tt><br>
<br>
So what other options should I try to
improve solver performance? Any
tips/insights would be appreciated as
preconditioning is black magic to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For reports, always run with </div>
<div><br>
</div>
<div> -ksp_view -ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that we can see exactly what you used.</div>
<div><br>
</div>
<div>I believe the default is a diagonal
factorization. Since your outer iterates are
increasing, I would strengthen this</div>
<div>to either upper or full</div>
<div><br>
</div>
<div> -pc_fieldsplit_schur_factorization_type
<upper, full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Thanks
in advance.<span><font color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all"><span><font color="#888888">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener </font></span></font></span></div><span><font color="#888888">
</font></span></div><span><font color="#888888">
</font></span></blockquote><span><font color="#888888">
<br>
</font></span></div><span><font color="#888888">
</font></span></blockquote><span><font color="#888888">
</font></span></div><span><font color="#888888">
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener</font></span></div>
</div>
</blockquote>
</div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener
</div></div></div></div>
</blockquote></div><br></div>