<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Matt<br>
<br>
The system is certainly not singular (checked using -pc_type svd
-pc_svd_monitor). It also gives the correct solution (compared to
the analytical solution for a trivial problem).<br>
<br>
Tabrez<br>
<br>
On 10/28/2014 08:44 AM, Matthew Knepley wrote:<br>
</div>
<blockquote
cite="mid:CAMYG4GkrxZAkBb9YUOkWCNJ362gdby3dvKUNjTruWO0-UePzFw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at 8:00 AM,
Tabrez Ali <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu" target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Matt<br>
<br>
With MUMPS it is indeed the same but not faster. E.g.,
see 1.txt and 2.txt<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>There is a big problem here. This system is never
actually being solved. Look at 1.txt:</div>
<div><br>
</div>
<div><span style="color:rgb(0,0,0);font-family:'Courier
New',Courier,monospace;font-size:14.4444446563721px;white-space:pre-wrap">
0 KSP preconditioned resid norm 9.320055451716e+05 true
resid norm 1.755998647494e+02 ||r(i)||/||b||
1.000000000000e+00
</span></div>
<div><span style="color:rgb(0,0,0);font-family:'Courier
New',Courier,monospace;font-size:14.4444446563721px;white-space:pre-wrap">380
KSP preconditioned resid norm 9.034871458425e-05 true
resid norm 3.118451181896e+09 ||r(i)||/||b||
1.775884728811e+07
</span></div>
<div><br>
</div>
<div>The preconditioned residual did drop 10 order of
magnitude, but the true residual (b - Ax) is still very
very large.</div>
<div>This looks like the system is singular and the
preconditioner is masking this. This needs to be figured
out first.</div>
<div>I would go back to a small serial problem, and make
sure it is actually being solved.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With GAMG, cg takes forever (see 4.txt). For
reference GAMG with preonly is attached again (3.txt)<br>
<br>
All logs include -fieldsplit_0_ksp_monitor.<br>
<br>
Also, the true residual shows small oscillations. Is
that normal?<br>
<br>
Tabrez<br>
<br>
<br>
On 10/28/2014 07:08 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Oct 28, 2014 at
5:39 AM, Tabrez Ali <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Mark<br>
<br>
When I replace "-fieldsplit_0_ksp_type
preonly" with "-fieldsplit_0_ksp_type cg"
then it becomes very slow (had to kill
it).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>That means something in the setup is wrong.
It should be about the same or faster. Run
with -fieldsplit_0_ksp_monitor</div>
<div>so we can see what is happening.</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> With MUMPS , i.e., with
'-fieldsplit_0_pc_type lu
-fieldsplit_0_pc_factor_mat_solver_package
mumps -fieldsplit_0_ksp_type preonly' it
works fine but takes more time, and will
be an issue for larger problems. The
output for this run is attached.<br>
<br>
I will work on passing rigid body modes
(as Matt mentioned) but short of that what
is the best set of options for solving the
following problem (i.e., linear elasticity
with constraints):<br>
<br>
<tt>|K cG'| | u | = |F|</tt><tt><br>
</tt><tt>|G 0 | |l/c| |d|</tt><br>
<br>
where c is a scaling factor (so that cG'
terms are more or less of the same order
as K)? The constraints are used to impose
slip between surfaces and so on.<br>
<br>
Tabrez<br>
<br>
On 10/27/2014 01:17 PM, Mark Adams wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">The null space for GAMG is
not critical but useful for elasticity.
If you in fact have an indefinite
operator (eg, not "pinned) the you need
to use an iterative coarse grid solver.
You are using '-fieldsplit_0_pc_type
gamg -fieldsplit_0_ksp_type preonly'.
And you have a hard elasticity problem.
You are going to want to start with a
stronger solver. Use cg instead of
preonly. As Matt said start with MUMPS,
then go to CG/GAMG, then you can see how
far you can cut the _0_ solver down.
<div>Mark</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Oct 23,
2014 at 11:51 AM, Matthew Knepley <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:knepley@gmail.com"
target="_blank">knepley@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span>On
Thu, Oct 23, 2014 at 10:48
AM, Tabrez Ali <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000"
bgcolor="#ffffff"> Matt<br>
<br>
On 10/23/2014 09:54 AM,
Matthew Knepley wrote:
<blockquote type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Thu, Oct 23,
2014 at 9:27 AM,
Tabrez Ali <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div>Matt<br>
<br>
Sorry about
that (I always
forget it).
The output for
the smallest
problem is now
attached (see
log.txt). I am
also attaching
some results
that compare
results
obtained using
FS/LSC and the
direct solver
(MUMPS), again
for the
smallest
problem. The
difference, as
you can see is
insignificant
O(1E-6).<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How do
you use MUMPS
if you have a
saddle point</div>
</div>
</div>
</div>
</blockquote>
I simply used -pc_type
lu
-pc_factor_mat_solver_package
mumps.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">
<div><br>
</div>
<div>2) You can
see from the
output that
something is
seriously
wrong with the
preconditioner.
It looks like
it has a null
space.</div>
<div> Did you
add the
elastic null
modes to GAMG?
Without this,
it is not
going to work.
We have helper
functions for
this:</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexCreateRigidBody.html</a></div>
<div><br>
</div>
<div>you could
just copy that
code. And then
use</div>
<div><br>
</div>
<div> <a
moz-do-not-send="true"
href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html"
target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetNearNullSpace.html</a></div>
<div><br>
</div>
<div>I don't see
it in the
output, so I
think this is
your problem.</div>
<div><br>
</div>
<div>In order to
test, I would
first use
MUMPS as the
A00 solver and
get the Schur
stuff worked
out. Then I
would</div>
<div>replace
MUMPS with
GAMG and tune
it until I get
back my
original
convergence.</div>
</div>
</div>
</div>
</blockquote>
I will try this with
MatNullSpaceCreateRigidBody.
Btw does it matter if
some nodes are pinned?<br>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>No these are null modes
of the operator, not of the
particular problem.</div>
<div><br>
</div>
<div> Matt</div>
<div>
<div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div text="#000000"
bgcolor="#ffffff">
Tabrez<br>
<br>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div> Also, I
did pass
'upper' and
'full' to
'-pc_fieldsplit_schur_factorization_type'
but the
iteration
count doesn't
improve (in
fact, it
increases
slightly). The
attached log
is with
'upper'.<br>
<br>
Regards,<br>
<br>
Tabrez<br>
<br>
On 10/23/2014
07:46 AM,
Matthew
Knepley wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div
class="gmail_extra">
<div
class="gmail_quote">On
Thu, Oct 23,
2014 at 7:20
AM, Tabrez Ali
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:stali@geology.wisc.edu"
target="_blank">stali@geology.wisc.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Hello<br>
<br>
I am using the
following
options
(below) for
solving linear
elasticity/poroelasticity
problems
involving slip
between two
surfaces
involving
non-trivial
geometries,
i.e., elements
with high
aspect ratios,
large
contrasts in
material
properties
etc. The
constraints
are imposed
using Lagrange
Multipliers. <br>
<br>
A picture
(shows
displacement
magnitude) is
attached. The
boundary
nodes, i.e.,
the base and
the four side
are pinned.<br>
<br>
The following
options appear
to work well
for the saddle
point problem:<br>
<br>
<tt>-pc_type
fieldsplit
-pc_fieldsplit_type
schur
-pc_fieldsplit_detect_saddle_point
-fieldsplit_0_pc_type
gamg
-fieldsplit_0_ksp_type
preonly
-fieldsplit_1_pc_type
lsc
-fieldsplit_1_ksp_type
preonly
-pc_fieldsplit_schur_fact_type
lower
-ksp_monitor</tt><br>
<br>
However, the
number of
iterations
keep on
increasing
with the
problems size
(see attached
plot), e.g.,<br>
<br>
<tt>120K
Tets <b>507</b>
Iterations
(KSP Residual
norm
8.827362494659e-05)</tt><tt>
in 17 secs on
3 cores<br>
</tt><tt>1
Million Tets
<b>1374</b>
Iterations
(KSP Residual
norm
7.164704416296e-05)</tt><tt>
in 117 secs
on 20 cores<br>
</tt><tt>8
Million Tets
<b>2495</b>
Iterations
(KSP Residual
norm
9.101247550026e-05)
in 225 secs on
160 cores</tt><br>
<br>
So what other
options should
I try to
improve solver
performance?
Any
tips/insights
would be
appreciated as
preconditioning
is black magic
to me.<br>
</div>
</blockquote>
<div><br>
</div>
<div>For
reports,
always run
with </div>
<div><br>
</div>
<div>
-ksp_view
-ksp_monitor_true_residual
-ksp_converged_reason</div>
<div><br>
</div>
<div>so that
we can see
exactly what
you used.</div>
<div><br>
</div>
<div>I believe
the default is
a diagonal
factorization.
Since your
outer iterates
are
increasing, I
would
strengthen
this</div>
<div>to either
upper or full</div>
<div><br>
</div>
<div>
-pc_fieldsplit_schur_factorization_type
<upper,
full></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div>
Matt </div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
Thanks in
advance.<span><font
color="#888888"><br>
<br>
Tabrez<br>
</font></span></div>
</blockquote>
</div>
<br>
<br
clear="all">
<span><font
color="#888888">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener </font></span></font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
<br>
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
<br>
<br
clear="all">
<span><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</font></span></font></span></div>
<span><font
color="#888888">
</font></span></div>
<span><font
color="#888888">
</font></span></blockquote>
<span><font
color="#888888"> </font></span></div>
<span><font
color="#888888"> </font></span></blockquote>
<span><font
color="#888888"> </font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888">
<div>
<div><br>
<br clear="all">
<span class=""><font
color="#888888">
<div><br>
</div>
-- <br>
What most
experimenters take
for granted before
they begin their
experiments is
infinitely more
interesting than any
results to which
their experiments
lead.<br>
-- Norbert Wiener </font></span></div>
<span class=""><font
color="#888888"> </font></span></div>
<span class=""><font
color="#888888"> </font></span></font></span></div>
<span class=""><font
color="#888888"> <span><font
color="#888888"> </font></span></font></span></div>
<span class=""><font color="#888888">
<span><font color="#888888"> </font></span></font></span></blockquote>
<span class=""><font color="#888888">
<span><font color="#888888"> </font></span></font></span></div>
<span class=""><font color="#888888"> <span><font
color="#888888"> <br>
</font></span></font></span></div>
<span class=""><font color="#888888"> </font></span></blockquote>
<span class=""><font color="#888888"> <br>
</font></span></div>
<span class=""><font color="#888888"> </font></span></blockquote>
<span class=""><font color="#888888"> </font></span></div>
<span class=""><font color="#888888"> <br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener </font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener
</div>
</div>
</blockquote>
<br>
</body>
</html>