<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Matt and Jed<br>
<br>
But it does appear to work well in practice (as in I get the
correct solution). Based on what I have tried, LSC (with GAMG on
00) is the only solver I am able to use on relatively large
problems with non trivial geometries (e.g., see my original
email). <br>
<br>
So why is that? <br>
<br>
I will experiment with your other suggestions.<br>
<br>
Thanks,<br>
<br>
Tabrez<br>
<br>
<br>
On 10/28/2014 09:55 AM, Matthew Knepley wrote:<br>
</div>
<blockquote
cite="mid:CAMYG4GnmDOi1-r1SBSLi3N+Sy1QhEevMTxmFjTU6+RGkmFw2yQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at 9:49 AM,
Tabrez Ali <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
The system is certainly not singular (checked using
-pc_type svd -pc_svd_monitor). It also gives the
correct solution (compared to the analytical solution
for a trivial problem).</div>
</div>
</blockquote>
<div><br>
</div>
<div>Something is very wrong with the preconditioner then.
Move back to full Schur complement with a tight tolerance</div>
<div>on the solver for S. This should converge in 1 iterate
with a good true residual. Perhaps there is something
wrong</div>
<div>with LSC for your system. It is not guaranteed to work
for all saddle points.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div><span class="HOEnZb"><font color="#888888"><br>
Tabrez</font></span>
<div>
<div class="h5"><br>
<br>
On 10/28/2014 08:44 AM, Matthew Knepley wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014
at 8:00 AM, Tabrez Ali <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
With MUMPS it is indeed the same but
not faster. E.g., see 1.txt and 2.txt<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>There is a big problem here. This
system is never actually being solved.
Look at 1.txt:</div>
<div><br>
</div>
<div><span> 0 KSP preconditioned resid norm
9.320055451716e+05 true resid norm
1.755998647494e+02 ||r(i)||/||b||
1.000000000000e+00 </span></div>
<div><span>380 KSP preconditioned resid norm
9.034871458425e-05 true resid norm
3.118451181896e+09 ||r(i)||/||b||
1.775884728811e+07 </span></div>
<div><br>
</div>
<div>The preconditioned residual did drop 10
order of magnitude, but the true residual
(b - Ax) is still very very large.</div>
<div>This looks like the system is singular
and the preconditioner is masking this.
This needs to be figured out first.</div>
<div>I would go back to a small serial
problem, and make sure it is actually
being solved.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With GAMG, cg takes forever (see
4.txt). For reference GAMG with
preonly is attached again (3.txt)<br>
<br>
All logs include
-fieldsplit_0_ksp_monitor.<br>
<br>
Also, the true residual shows small
oscillations. Is that normal?<br>
<br>
Tabrez<br>
<br>
<br>
On 10/28/2014 07:08 AM, Matthew
Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue,
Oct 28, 2014 at 5:39 AM, Tabrez
Ali <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<div>Mark<br>
<br>
When I replace
"-fieldsplit_0_ksp_type
preonly" with
"-fieldsplit_0_ksp_type
cg" then it becomes very
slow (had to kill it).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>That means something in the
setup is wrong. It should be
about the same or faster. Run
with -fieldsplit_0_ksp_monitor</div>
<div>so we can see what is
happening.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<div> With MUMPS , i.e.,
with
'-fieldsplit_0_pc_type lu
-fieldsplit_0_pc_factor_mat_solver_package
mumps
-fieldsplit_0_ksp_type
preonly' it works fine but
takes more time, and will
be an issue for larger
problems. The output for
this run is attached.<br>
<br>
I will work on passing
rigid body modes (as Matt
mentioned) but short of
that what is the best set
of options for solving the
following problem (i.e.,
linear elasticity with
constraints):<br>
<br>
<tt>|K cG'| | u | = |F|</tt><tt><br>
</tt><tt>|G 0 | |l/c|
|d|</tt><br>
<br>
where c is a scaling
factor (so that cG' terms
are more or less of the
same order as K)? The
constraints are used to
impose slip between
surfaces and so on.<br>
<br>
Tabrez<br>
<br>
On 10/27/2014 01:17 PM,
Mark Adams wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">The null
space for GAMG is not
critical but useful for
elasticity. If you in
fact have an indefinite
operator (eg, not
"pinned) the you need
to use an iterative
coarse grid solver. You
are
using '-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly'. And you have
a hard elasticity
problem. You are going
to want to start with a
stronger solver. Use cg
instead of preonly. As
Matt said start with
MUMPS, then go to
CG/GAMG, then you can
see how far you can cut
the _0_ solver down.
<div>Mark</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Oct 23, 2014 at
11:51 AM, Matthew
Knepley <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote"><span>On
Thu, Oct 23,
2014 at 10:48
AM, Tabrez Ali
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#ffffff"> Matt<br>
<br>
On 10/23/2014
09:54 AM,
Matthew
Knepley wrote:
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Thu, Oct 23,
2014 at 9:27
AM, Tabrez Ali
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div>Matt<br>
<br>
Sorry about
that (I always
forget it).
The output for
the smallest
problem is now
attached (see
log.txt). I am
also attaching
some results
that compare
results
obtained using
FS/LSC and the
direct solver
(MUMPS), again
for the
smallest
problem. The
difference, as
you can see is
insignificant
O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do
you use MUMPS
if you have a
saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used
-pc_type lu
-pc_factor_mat_solver_package
mumps.<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">
<div><br>
</div>
<div>2) You
can see from
the output
that something
is seriously
wrong with the
preconditioner.
It looks like
it has a null
space.</div>
<div> Did
you add the
elastic null
modes to GAMG?
Without this,
it is not
going to work.
We have helper
functions for
this:</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could
just copy that
code. And then
use</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't
see it in the
output, so I
think this is
your problem.</div>
<div><br>
</div>
<div>In order
to test, I
would first
use MUMPS as
the A00 solver
and get the
Schur stuff
worked out.
Then I would</div>
<div>replace
MUMPS with
GAMG and tune
it until I get
back my
original
convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try
this with
MatNullSpaceCreateRigidBody.
Btw does it
matter if some
nodes are
pinned?<br>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>No these
are null modes
of the
operator, not
of the
particular
problem.</div>
<div><br>
</div>
<div> Matt</div>
<div>
<div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#ffffff"> Tabrez<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div> Also, I
did pass
'upper' and
'full' to
'-pc_fieldsplit_schur_factorization_type'
but the
iteration
count doesn't
improve (in
fact, it
increases
slightly). The
attached log
is with
'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014
07:46 AM,
Matthew
Knepley wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Thu, Oct 23,
2014 at 7:20
AM, Tabrez Ali
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Hello<br>
<br>
I am using the
following
options
(below) for
solving linear
elasticity/poroelasticity
problems
involving slip
between two
surfaces
involving
non-trivial
geometries,
i.e., elements
with high
aspect ratios,
large
contrasts in
material
properties
etc. The
constraints
are imposed
using Lagrange
Multipliers. <br>
<br>
A picture
(shows
displacement
magnitude) is
attached. The
boundary
nodes, i.e.,
the base and
the four side
are pinned.<br>
<br>
The following
options appear
to work well
for the saddle
point problem:<br>
<br>
<tt>-pc_type
fieldsplit
-pc_fieldsplit_type
schur
-pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly
-fieldsplit_1_pc_type
lsc
-fieldsplit_1_ksp_type
preonly
-pc_fieldsplit_schur_fact_type
lower
-ksp_monitor</tt><br>
<br>
However, the
number of
iterations
keep on
increasing
with the
problems size
(see attached
plot), e.g.,<br>
<br>
<tt>120K
Tets <b>507</b>
Iterations
(KSP Residual
norm
8.827362494659e-05)</tt><tt>
in 17 secs on
3 cores<br>
</tt><tt>1
Million Tets
<b>1374</b>
Iterations
(KSP Residual
norm
7.164704416296e-05)</tt><tt>
in 117 secs
on 20 cores<br>
</tt><tt>8
Million Tets
<b>2495</b>
Iterations
(KSP Residual
norm
9.101247550026e-05)
in 225 secs on
160 cores</tt><br>
<br>
So what other
options should
I try to
improve solver
performance?
Any
tips/insights
would be
appreciated as
preconditioning
is black magic
to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For
reports,
always run
with </div>
<div><br>
</div>
<div>
-ksp_view
-ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that
we can see
exactly what
you used.</div>
<div><br>
</div>
<div>I believe
the default is
a diagonal
factorization.
Since your
outer iterates
are
increasing, I
would
strengthen
this</div>
<div>to either
upper or full</div>
<div><br>
</div>
<div>
-pc_fieldsplit_schur_factorization_type
<upper,
full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt </div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Thanks in
advance.<span><font
color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br
clear="all">
<span><font
color="#888888">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener </font></span></font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
<br>
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
<br>
<br
clear="all">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</font></span></font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
<div>
<div><br>
<br
clear="all">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener </font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></font></span></div>
<span><font
color="#888888">
<span><font
color="#888888">
</font></span></font></span></div>
<span><font
color="#888888">
<span><font
color="#888888">
</font></span></font></span></blockquote>
<span><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span><font
color="#888888"> <span><font
color="#888888">
<br>
</font></span></font></span></div>
<span><font
color="#888888"> </font></span></blockquote>
<span><font color="#888888">
<br>
</font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> <br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take
for granted before they begin
their experiments is
infinitely more interesting
than any results to which
their experiments lead.<br>
-- Norbert Wiener </font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener </div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener
</div>
</div>
</blockquote>
<br>
</body>
</html>