<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Mark<br>
<br>
When I replace "-fieldsplit_0_ksp_type preonly" with
"-fieldsplit_0_ksp_type cg" then it becomes very slow (had to kill
it).<br>
<br>
With MUMPS , i.e., with '-fieldsplit_0_pc_type lu
-fieldsplit_0_pc_factor_mat_solver_package mumps
-fieldsplit_0_ksp_type preonly' it works fine but takes more time,
and will be an issue for larger problems. The output for this run
is attached.<br>
<br>
I will work on passing rigid body modes (as Matt mentioned) but
short of that what is the best set of options for solving the
following problem (i.e., linear elasticity with constraints):<br>
<br>
<tt>|K cG'| | u | = |F|</tt><tt><br>
</tt><tt>|G 0 | |l/c| |d|</tt><br>
<br>
where c is a scaling factor (so that cG' terms are more or less of
the same order as K)? The constraints are used to impose slip
between surfaces and so on.<br>
<br>
Tabrez<br>
<br>
On 10/27/2014 01:17 PM, Mark Adams wrote:<br>
</div>
<blockquote
cite="mid:CADOhEh7RwJgGdofn-xCx5V944dwZXFaHZfqxZEqjGdjvC06Ckg@mail.gmail.com"
type="cite">
<div dir="ltr">The null space for GAMG is not critical but useful
for elasticity. If you in fact have an indefinite operator (eg,
not "pinned) the you need to use an iterative coarse grid
solver. You are using '-fieldsplit_0_pc_type gamg
-fieldsplit_0_ksp_type preonly'. And you have a hard elasticity
problem. You are going to want to start with a stronger solver.
Use cg instead of preonly. As Matt said start with MUMPS, then
go to CG/GAMG, then you can see how far you can cut the _0_
solver down.
<div>Mark</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 23, 2014 at 11:51 AM,
Matthew Knepley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span class="">On Thu, Oct 23,
2014 at 10:48 AM, Tabrez Ali <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> Matt<br>
<br>
On 10/23/2014 09:54 AM, Matthew Knepley wrote:
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Oct 23,
2014 at 9:27 AM, Tabrez Ali <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
Sorry about that (I always forget
it). The output for the smallest
problem is now attached (see
log.txt). I am also attaching some
results that compare results
obtained using FS/LSC and the
direct solver (MUMPS), again for
the smallest problem. The
difference, as you can see is
insignificant O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do you use MUMPS if you have
a saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used -pc_type lu
-pc_factor_mat_solver_package mumps.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>2) You can see from the output that
something is seriously wrong with the
preconditioner. It looks like it has a
null space.</div>
<div> Did you add the elastic null
modes to GAMG? Without this, it is not
going to work. We have helper
functions for this:</div>
<div><br>
</div>
<div> <a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could just copy that code. And
then use</div>
<div><br>
</div>
<div> <a moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't see it in the output, so I
think this is your problem.</div>
<div><br>
</div>
<div>In order to test, I would first use
MUMPS as the A00 solver and get the
Schur stuff worked out. Then I would</div>
<div>replace MUMPS with GAMG and tune it
until I get back my original
convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try this with
MatNullSpaceCreateRigidBody. Btw does it matter
if some nodes are pinned?<br>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>No these are null modes of the operator, not of
the particular problem.</div>
<div><br>
</div>
<div> Matt</div>
<div>
<div class="h5">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> Tabrez<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<div> Also, I did pass 'upper' and
'full' to
'-pc_fieldsplit_schur_factorization_type'
but the iteration count doesn't
improve (in fact, it increases
slightly). The attached log is
with 'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014 07:46 AM, Matthew
Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Thu, Oct 23, 2014 at 7:20
AM, Tabrez Ali <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Hello<br>
<br>
I am using the
following options
(below) for solving
linear
elasticity/poroelasticity
problems involving
slip between two
surfaces involving
non-trivial
geometries, i.e.,
elements with high
aspect ratios, large
contrasts in material
properties etc. The
constraints are
imposed using Lagrange
Multipliers. <br>
<br>
A picture (shows
displacement
magnitude) is
attached. The boundary
nodes, i.e., the base
and the four side are
pinned.<br>
<br>
The following options
appear to work well
for the saddle point
problem:<br>
<br>
<tt>-pc_type
fieldsplit
-pc_fieldsplit_type
schur
-pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly
-fieldsplit_1_pc_type
lsc
-fieldsplit_1_ksp_type
preonly
-pc_fieldsplit_schur_fact_type
lower -ksp_monitor</tt><br>
<br>
However, the number of
iterations keep on
increasing with the
problems size (see
attached plot), e.g.,<br>
<br>
<tt>120K Tets <b>507</b>
Iterations (KSP
Residual norm
8.827362494659e-05)</tt><tt>
in 17 secs on 3
cores<br>
</tt><tt>1 Million
Tets <b>1374</b>
Iterations (KSP
Residual norm
7.164704416296e-05)</tt><tt>
in 117 secs on 20
cores<br>
</tt><tt>8 Million
Tets <b>2495</b>
Iterations (KSP
Residual norm
9.101247550026e-05)
in 225 secs on 160
cores</tt><br>
<br>
So what other options
should I try to
improve solver
performance? Any
tips/insights would be
appreciated as
preconditioning is
black magic to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For reports, always
run with </div>
<div><br>
</div>
<div> -ksp_view
-ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that we can see
exactly what you used.</div>
<div><br>
</div>
<div>I believe the default
is a diagonal
factorization. Since
your outer iterates are
increasing, I would
strengthen this</div>
<div>to either upper or
full</div>
<div><br>
</div>
<div>
-pc_fieldsplit_schur_factorization_type
<upper, full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> Thanks
in advance.<span><font
color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters take
for granted before
they begin their
experiments is
infinitely more
interesting than any
results to which
their experiments
lead.<br>
-- Norbert Wiener </font></span></font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
</font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> <br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</font></span></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<div class="h5"><br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they
begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>