<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 28, 2014 at 8:00 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
With MUMPS it is indeed the same but not faster. E.g., see 1.txt
and 2.txt<br></div></div></blockquote><div><br></div><div>There is a big problem here. This system is never actually being solved. Look at 1.txt:</div><div><br></div><div><span style="color:rgb(0,0,0);font-family:'Courier New',Courier,monospace;font-size:14.4444446563721px;white-space:pre-wrap"> 0 KSP preconditioned resid norm 9.320055451716e+05 true resid norm 1.755998647494e+02 ||r(i)||/||b|| 1.000000000000e+00
</span></div><div><span style="color:rgb(0,0,0);font-family:'Courier New',Courier,monospace;font-size:14.4444446563721px;white-space:pre-wrap">380 KSP preconditioned resid norm 9.034871458425e-05 true resid norm 3.118451181896e+09 ||r(i)||/||b|| 1.775884728811e+07
</span></div><div><br></div><div>The preconditioned residual did drop 10 order of magnitude, but the true residual (b - Ax) is still very very large.</div><div>This looks like the system is singular and the preconditioner is masking this. This needs to be figured out first.</div><div>I would go back to a small serial problem, and make sure it is actually being solved.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
With GAMG, cg takes forever (see 4.txt). For reference GAMG with
preonly is attached again (3.txt)<br>
<br>
All logs include -fieldsplit_0_ksp_monitor.<br>
<br>
Also, the true residual shows small oscillations. Is that normal?<br>
<br>
Tabrez<br>
<br>
<br>
On 10/28/2014 07:08 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at 5:39 AM,
Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Mark<br>
<br>
When I replace "-fieldsplit_0_ksp_type preonly" with
"-fieldsplit_0_ksp_type cg" then it becomes very slow
(had to kill it).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>That means something in the setup is wrong. It should
be about the same or faster. Run with
-fieldsplit_0_ksp_monitor</div>
<div>so we can see what is happening.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With MUMPS , i.e., with '-fieldsplit_0_pc_type lu
-fieldsplit_0_pc_factor_mat_solver_package mumps
-fieldsplit_0_ksp_type preonly' it works fine but
takes more time, and will be an issue for larger
problems. The output for this run is attached.<br>
<br>
I will work on passing rigid body modes (as Matt
mentioned) but short of that what is the best set of
options for solving the following problem (i.e.,
linear elasticity with constraints):<br>
<br>
<tt>|K cG'| | u | = |F|</tt><tt><br>
</tt><tt>|G 0 | |l/c| |d|</tt><br>
<br>
where c is a scaling factor (so that cG' terms are
more or less of the same order as K)? The constraints
are used to impose slip between surfaces and so on.<br>
<br>
Tabrez<br>
<br>
On 10/27/2014 01:17 PM, Mark Adams wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">The null space for GAMG is not critical
but useful for elasticity. If you in fact have an
indefinite operator (eg, not "pinned) the you need
to use an iterative coarse grid solver. You are
using '-fieldsplit_0_pc_type gamg
-fieldsplit_0_ksp_type preonly'. And you have a
hard elasticity problem. You are going to want to
start with a stronger solver. Use cg instead of
preonly. As Matt said start with MUMPS, then go to
CG/GAMG, then you can see how far you can cut the
_0_ solver down.
<div>Mark</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 23, 2014 at
11:51 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span>On Thu, Oct
23, 2014 at 10:48 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
Matt<br>
<br>
On 10/23/2014 09:54 AM, Matthew
Knepley wrote:
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Thu, Oct 23, 2014 at 9:27
AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
Sorry about that (I
always forget it). The
output for the
smallest problem is
now attached (see
log.txt). I am also
attaching some results
that compare results
obtained using FS/LSC
and the direct solver
(MUMPS), again for the
smallest problem. The
difference, as you can
see is insignificant
O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do you use MUMPS
if you have a saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used -pc_type lu
-pc_factor_mat_solver_package mumps.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>2) You can see from the
output that something is
seriously wrong with the
preconditioner. It looks
like it has a null space.</div>
<div> Did you add the
elastic null modes to
GAMG? Without this, it is
not going to work. We have
helper functions for this:</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could just copy
that code. And then use</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't see it in the
output, so I think this is
your problem.</div>
<div><br>
</div>
<div>In order to test, I
would first use MUMPS as
the A00 solver and get the
Schur stuff worked out.
Then I would</div>
<div>replace MUMPS with GAMG
and tune it until I get
back my original
convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try this with
MatNullSpaceCreateRigidBody. Btw
does it matter if some nodes are
pinned?<br>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>No these are null modes of the
operator, not of the particular problem.</div>
<div><br>
</div>
<div> Matt</div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> Tabrez<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> Also, I did pass
'upper' and 'full'
to
'-pc_fieldsplit_schur_factorization_type'
but the iteration
count doesn't
improve (in fact, it
increases slightly).
The attached log is
with 'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014 07:46
AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Thu, Oct 23,
2014 at 7:20
AM, Tabrez Ali
<span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello<br>
<br>
I am using the
following
options
(below) for
solving linear
elasticity/poroelasticity
problems
involving slip
between two
surfaces
involving
non-trivial
geometries,
i.e., elements
with high
aspect ratios,
large
contrasts in
material
properties
etc. The
constraints
are imposed
using Lagrange
Multipliers. <br>
<br>
A picture
(shows
displacement
magnitude) is
attached. The
boundary
nodes, i.e.,
the base and
the four side
are pinned.<br>
<br>
The following
options appear
to work well
for the saddle
point problem:<br>
<br>
<tt>-pc_type
fieldsplit
-pc_fieldsplit_type
schur
-pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly
-fieldsplit_1_pc_type
lsc
-fieldsplit_1_ksp_type
preonly
-pc_fieldsplit_schur_fact_type
lower
-ksp_monitor</tt><br>
<br>
However, the
number of
iterations
keep on
increasing
with the
problems size
(see attached
plot), e.g.,<br>
<br>
<tt>120K
Tets <b>507</b>
Iterations
(KSP Residual
norm
8.827362494659e-05)</tt><tt>
in 17 secs on
3 cores<br>
</tt><tt>1
Million Tets
<b>1374</b>
Iterations
(KSP Residual
norm
7.164704416296e-05)</tt><tt>
in 117 secs
on 20 cores<br>
</tt><tt>8
Million Tets
<b>2495</b>
Iterations
(KSP Residual
norm
9.101247550026e-05)
in 225 secs on
160 cores</tt><br>
<br>
So what other
options should
I try to
improve solver
performance?
Any
tips/insights
would be
appreciated as
preconditioning
is black magic
to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For
reports,
always run
with </div>
<div><br>
</div>
<div>
-ksp_view
-ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that
we can see
exactly what
you used.</div>
<div><br>
</div>
<div>I believe
the default is
a diagonal
factorization.
Since your
outer iterates
are
increasing, I
would
strengthen
this</div>
<div>to either
upper or full</div>
<div><br>
</div>
<div>
-pc_fieldsplit_schur_factorization_type
<upper,
full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Thanks in
advance.<span><font color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener </font></span></font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888"> <br>
</font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888">
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters take
for granted before
they begin their
experiments is
infinitely more
interesting than any
results to which
their experiments
lead.<br>
-- Norbert Wiener</font></span></font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
<div>
<div><br>
<br clear="all"><span class=""><font color="#888888">
<div><br>
</div>
-- <br>
What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener </font></span></div><span class=""><font color="#888888">
</font></span></div><span class=""><font color="#888888">
</font></span></font></span></div><span class=""><font color="#888888">
<span><font color="#888888"> </font></span></font></span></div><span class=""><font color="#888888">
<span><font color="#888888"> </font></span></font></span></blockquote><span class=""><font color="#888888">
<span><font color="#888888"> </font></span></font></span></div><span class=""><font color="#888888">
<span><font color="#888888"> <br>
</font></span></font></span></div><span class=""><font color="#888888">
</font></span></blockquote><span class=""><font color="#888888">
<br>
</font></span></div><span class=""><font color="#888888">
</font></span></blockquote><span class=""><font color="#888888">
</font></span></div><span class=""><font color="#888888">
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener
</div></div>