<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 28, 2014 at 9:49 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
The system is certainly not singular (checked using -pc_type svd
-pc_svd_monitor). It also gives the correct solution (compared to
the analytical solution for a trivial problem).</div></div></blockquote><div><br></div><div>Something is very wrong with the preconditioner then. Move back to full Schur complement with a tight tolerance</div><div>on the solver for S. This should converge in 1 iterate with a good true residual. Perhaps there is something wrong</div><div>with LSC for your system. It is not guaranteed to work for all saddle points.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><span class="HOEnZb"><font color="#888888"><br>
Tabrez</font></span><div><div class="h5"><br>
<br>
On 10/28/2014 08:44 AM, Matthew Knepley wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at 8:00 AM,
Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
With MUMPS it is indeed the same but not faster. E.g.,
see 1.txt and 2.txt<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>There is a big problem here. This system is never
actually being solved. Look at 1.txt:</div>
<div><br>
</div>
<div><span>
0 KSP preconditioned resid norm 9.320055451716e+05 true
resid norm 1.755998647494e+02 ||r(i)||/||b||
1.000000000000e+00
</span></div>
<div><span>380
KSP preconditioned resid norm 9.034871458425e-05 true
resid norm 3.118451181896e+09 ||r(i)||/||b||
1.775884728811e+07
</span></div>
<div><br>
</div>
<div>The preconditioned residual did drop 10 order of
magnitude, but the true residual (b - Ax) is still very
very large.</div>
<div>This looks like the system is singular and the
preconditioner is masking this. This needs to be figured
out first.</div>
<div>I would go back to a small serial problem, and make
sure it is actually being solved.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With GAMG, cg takes forever (see 4.txt). For
reference GAMG with preonly is attached again (3.txt)<br>
<br>
All logs include -fieldsplit_0_ksp_monitor.<br>
<br>
Also, the true residual shows small oscillations. Is
that normal?<br>
<br>
Tabrez<br>
<br>
<br>
On 10/28/2014 07:08 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at
5:39 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Mark<br>
<br>
When I replace "-fieldsplit_0_ksp_type
preonly" with "-fieldsplit_0_ksp_type cg"
then it becomes very slow (had to kill
it).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>That means something in the setup is wrong.
It should be about the same or faster. Run
with -fieldsplit_0_ksp_monitor</div>
<div>so we can see what is happening.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With MUMPS , i.e., with
'-fieldsplit_0_pc_type lu
-fieldsplit_0_pc_factor_mat_solver_package
mumps -fieldsplit_0_ksp_type preonly' it
works fine but takes more time, and will
be an issue for larger problems. The
output for this run is attached.<br>
<br>
I will work on passing rigid body modes
(as Matt mentioned) but short of that what
is the best set of options for solving the
following problem (i.e., linear elasticity
with constraints):<br>
<br>
<tt>|K cG'| | u | = |F|</tt><tt><br>
</tt><tt>|G 0 | |l/c| |d|</tt><br>
<br>
where c is a scaling factor (so that cG'
terms are more or less of the same order
as K)? The constraints are used to impose
slip between surfaces and so on.<br>
<br>
Tabrez<br>
<br>
On 10/27/2014 01:17 PM, Mark Adams wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">The null space for GAMG is
not critical but useful for elasticity.
If you in fact have an indefinite
operator (eg, not "pinned) the you need
to use an iterative coarse grid solver.
You are using '-fieldsplit_0_pc_type
gamg -fieldsplit_0_ksp_type preonly'.
And you have a hard elasticity problem.
You are going to want to start with a
stronger solver. Use cg instead of
preonly. As Matt said start with MUMPS,
then go to CG/GAMG, then you can see how
far you can cut the _0_ solver down.
<div>Mark</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 23,
2014 at 11:51 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span>On
Thu, Oct 23, 2014 at 10:48
AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> Matt<br>
<br>
On 10/23/2014 09:54 AM,
Matthew Knepley wrote:
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Thu, Oct 23,
2014 at 9:27 AM,
Tabrez Ali <span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
Sorry about
that (I always
forget it).
The output for
the smallest
problem is now
attached (see
log.txt). I am
also attaching
some results
that compare
results
obtained using
FS/LSC and the
direct solver
(MUMPS), again
for the
smallest
problem. The
difference, as
you can see is
insignificant
O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do
you use MUMPS
if you have a
saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used -pc_type
lu
-pc_factor_mat_solver_package
mumps.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>2) You can
see from the
output that
something is
seriously
wrong with the
preconditioner.
It looks like
it has a null
space.</div>
<div> Did you
add the
elastic null
modes to GAMG?
Without this,
it is not
going to work.
We have helper
functions for
this:</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could
just copy that
code. And then
use</div>
<div><br>
</div>
<div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't see
it in the
output, so I
think this is
your problem.</div>
<div><br>
</div>
<div>In order to
test, I would
first use
MUMPS as the
A00 solver and
get the Schur
stuff worked
out. Then I
would</div>
<div>replace
MUMPS with
GAMG and tune
it until I get
back my
original
convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try this with
MatNullSpaceCreateRigidBody.
Btw does it matter if
some nodes are pinned?<br>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>No these are null modes
of the operator, not of the
particular problem.</div>
<div><br>
</div>
<div> Matt</div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
Tabrez<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> Also, I
did pass
'upper' and
'full' to
'-pc_fieldsplit_schur_factorization_type'
but the
iteration
count doesn't
improve (in
fact, it
increases
slightly). The
attached log
is with
'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014
07:46 AM,
Matthew
Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On
Thu, Oct 23,
2014 at 7:20
AM, Tabrez Ali
<span dir="ltr"><<a href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hello<br>
<br>
I am using the
following
options
(below) for
solving linear
elasticity/poroelasticity
problems
involving slip
between two
surfaces
involving
non-trivial
geometries,
i.e., elements
with high
aspect ratios,
large
contrasts in
material
properties
etc. The
constraints
are imposed
using Lagrange
Multipliers. <br>
<br>
A picture
(shows
displacement
magnitude) is
attached. The
boundary
nodes, i.e.,
the base and
the four side
are pinned.<br>
<br>
The following
options appear
to work well
for the saddle
point problem:<br>
<br>
<tt>-pc_type
fieldsplit
-pc_fieldsplit_type
schur
-pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly
-fieldsplit_1_pc_type
lsc
-fieldsplit_1_ksp_type
preonly
-pc_fieldsplit_schur_fact_type
lower
-ksp_monitor</tt><br>
<br>
However, the
number of
iterations
keep on
increasing
with the
problems size
(see attached
plot), e.g.,<br>
<br>
<tt>120K
Tets <b>507</b>
Iterations
(KSP Residual
norm
8.827362494659e-05)</tt><tt>
in 17 secs on
3 cores<br>
</tt><tt>1
Million Tets
<b>1374</b>
Iterations
(KSP Residual
norm
7.164704416296e-05)</tt><tt>
in 117 secs
on 20 cores<br>
</tt><tt>8
Million Tets
<b>2495</b>
Iterations
(KSP Residual
norm
9.101247550026e-05)
in 225 secs on
160 cores</tt><br>
<br>
So what other
options should
I try to
improve solver
performance?
Any
tips/insights
would be
appreciated as
preconditioning
is black magic
to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For
reports,
always run
with </div>
<div><br>
</div>
<div>
-ksp_view
-ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that
we can see
exactly what
you used.</div>
<div><br>
</div>
<div>I believe
the default is
a diagonal
factorization.
Since your
outer iterates
are
increasing, I
would
strengthen
this</div>
<div>to either
upper or full</div>
<div><br>
</div>
<div>
-pc_fieldsplit_schur_factorization_type
<upper,
full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Thanks in
advance.<span><font color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<span><font color="#888888">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener </font></span></font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888">
<br>
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
<br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</font></span></font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888">
<div>
<div><br>
<br clear="all">
<span><font color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters take
for granted before
they begin their
experiments is
infinitely more
interesting than any
results to which
their experiments
lead.<br>
-- Norbert Wiener </font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></font></span></div>
<span><font color="#888888"> <span><font color="#888888"> </font></span></font></span></div>
<span><font color="#888888">
<span><font color="#888888"> </font></span></font></span></blockquote>
<span><font color="#888888">
<span><font color="#888888"> </font></span></font></span></div>
<span><font color="#888888"> <span><font color="#888888"> <br>
</font></span></font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
</font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> <br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener </font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener
</div></div>