<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">Le 17/05/2022 à 12:03, Dave May a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAJ98EDpWN7M9OWp7hvweTzb8JXBT3gG_YBUOxPP_mTFXUgwNOg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div><br>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Tue 17. May 2022 at 11:56, Loic Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu"
moz-do-not-send="true" class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Dave,</p>
<p>could you explain what you mean by state ?</p>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto">Ah - by state Matt and I mean any auxiliary
data stored in the user context passed to
KSPSetComputeOperators which is required to define your
operator and is distributed. For example, a Vec which is
used store coefficients of your PDE.</div>
<div dir="auto"><br>
</div>
</div>
</div>
</blockquote>
<p>Yes, I have a context which gives the operators (mult and
diagonal) and some other informations to build the matrix free on
each multigrid level.</p>
<p>You can find the idea here:
<a class="moz-txt-link-freetext" href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91</a></p>
<p>It's the definition of the operator used inside
DMKSPComputeOperators.</p>
<p>Loic<br>
</p>
<blockquote type="cite"
cite="mid:CAJ98EDpWN7M9OWp7hvweTzb8JXBT3gG_YBUOxPP_mTFXUgwNOg@mail.gmail.com">
<div>
<div class="gmail_quote">
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<div>Le 17/05/2022 à 11:50, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div>
<div dir="auto">Hi Loic,</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Can you confirm if your problem has
stored state which needs to be repartitioned?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Dave</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Tue 17. May 2022 at 09:07, Loic
Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div>Le 17/05/2022 à 01:43, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue,
17 May 2022 at 00:12, Matthew Knepley
<<a href="mailto:knepley@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, May 16, 2022 at
9:59 AM Loic Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Le 16/05/2022 à 21:54, Barry
Smith a écrit :<br>
</p>
<blockquote type="cite"> <br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at
3:50 PM, Loic Gouarin
<<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Thanks Barry for
your quick reply.<br>
</p>
<div>Le 16/05/2022 à
21:41, Barry Smith a
écrit :<br>
</div>
<blockquote
type="cite">
<div><br>
</div>
Loic,
<div><br>
</div>
<div> From your
code it looks like
you are using a
DM. Is it a DMDA
or a shell DM? If
it is a DMDA then
the process is
intended to be
pretty
straightforward.
PCTELESCOPE should
create a new DMDA
that has the same
properties as the
coarse grid DMDA
but lives on a
smaller number of
ranks. From this
you can just
provide multiple
levels of
coarsening of the
DMDA to produce
the smaller
multigrid
problems. <br>
</div>
</blockquote>
It's a DMDA. So what
you mean is to take
the KSP of
PCTELESCOPE, get the
DM which is the same
of the coarse grid and
build all the levels
as previously for the
levels of PCTELESCOPE
?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
Yes, I think you should be
able to pull out of the
coarse level PC (on the
subset of ranks) a DM that
and then setup coarse levels
from that. But you need to
look at an example that that
uses telescope and probably
even the telescope code
itself to see all the
details. I don't know them
anymore.</div>
</blockquote>
<p>I looked at the source code
but It didn't help. I will try
to explain more clearly my
experiments to understand what
I make wrong and you can help
more easily.</p>
</div>
</blockquote>
<div>Hi Loic!</div>
<div><br>
</div>
<div>Here is my guess. When
TELESCOPE redistributes the coarse
grid DMDA, it creates a new DMDA
and copies over all the
information it can see. However, I
think you may have state that is
attached to the old DMDA that is
not getting redistributed. If this
is true, I can help you write a
hook that redistributes that state
when TELESCOPE maps between
distributions.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you have state you need to
propagate between different
communicators, you will have to use the
telescope type which is internally is
referred to as <br>
<pre width="80">TELESCOPE_COARSEDM</pre>
</div>
<div><br>
</div>
<div>To activate this type you need the
option <br>
<pre width="80"><font color="#666666">-pc_telescope_use_coarse_dm</font></pre>
</div>
<div>Or to call
PCTelescopeSetUseCoarseDM()</div>
<div>The assumptions of usng this options
are described here</div>
<div><a
href="https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM</a></div>
<div><br>
</div>
<div>and source associated with the
COARSEDM <br>
</div>
<div><a
href="https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html</a></div>
<div><br>
</div>
<div>Using TELESCOPE_COARSEDM you will
have access to the hooks Matt is
referring to.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<p>Thanks Matthew and Dave.</p>
<p>I used TELESCOPE_COARSEDM in my previous
tests but it tells me that "Zero instances of
a coarse DM were found". And indeed, my coarse
solver only has the DM on the coarse mesh and
not on the fine level with a coarse DM
atttached. I think it's because I use
DMKSPSetComputeOperators to create the KSP on
each level.</p>
<p>So for now, I just added <br>
</p>
<div style="color:#d4d4d4;background-color:#1e1e1e;font-family:Menlo,Monaco,'Courier New',monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:#d4d4d4"> KSP coarse;</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCMGGetCoarseSolve</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_i</span><span style="color:#d4d4d4">, </span><span style="color:#b4b4b4">&</span><span style="color:#d4d4d4">coarse);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> PC </span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">;</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">KSPGetPC</span><span style="color:#d4d4d4">(coarse, </span><span style="color:#b4b4b4">&</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCSetType</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">, PCTELESCOPE);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCTelescopeSetUseCoarseDM</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">, PETSC_TRUE);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div></div>
<p>but it's not enough.</p>
<p>Cheers,</p>
<p>Loic<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>Cheers,</div>
<div>Dave<br>
</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF">
<blockquote
type="cite">
<div><br>
</div>
<div> Can you let
us know more
details of what
exactly goes wrong
with attempts? It
is likely
straightforward to
get things
working, perhaps
due to our unclear
documentation.</div>
</blockquote>
<p>I essentially have
wrong state or NULL
matrices into the
PCTELESCOPE.</p>
<p>Loic<br>
</p>
<blockquote
type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<div><br>
<blockquote
type="cite">
<div>On May
16, 2022, at
3:28 PM, Loic
Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div
bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I could
have posted my
message on the
user list but
it seems to me
that it's more
in the petsc
development
part. Don't
hesitate to
tell me if I'm
wrong.<br>
I am
developing a
code called
cafes that
studies
fluid-particle
interactions
in a Stokes
fluid. For
that, we
implemented
the whole
solver in
matrix free on
a cartesian
grid. The
preconditioner
is naturally a
geometrical
multigrid for
the velocity.
The set of
matrix free
for each level
is set by hand
(I worked with
Matthew on
this a long
time ago
now!). <br>
I would have
liked to
implement
telescope on
this part. I
started it but
I didn't get
out of it. So
I would like
to have your
help.<br>
What I tried
to do was to
describe my
classical
multigrid up
to a level N1
and then take
the coarse
solver and
apply the
telescope
preconditioner
to it up to a
level N2. I
don't know if
this is the
best way but
it seemed the
most
intuitive. The
problem is
that I could
not get the
coarse matrix
because the
setup is not
done yet (I
use a
DMKSPSetComputeOperators
for the
matrices). <br>
</p>
<p>The
construction
of the matrix
free for each
level is here:
<a
href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88</a></p>
<p>The
description of
the Stokes
preconditioner
is described
here : <a
href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162"
target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162</a></p>
<p>I have
tried multiple
implementations without success. For me, the start is to add at line 105
something like</p>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(87,166,74)">KSP coarse;</span></div><div><span style="color:rgb(87,166,74)">ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);</span></div><span style="color:rgb(87,166,74)">
</span></div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><span style="color:rgb(87,166,74)">PC pc_coarse;</span><div><span style="color:rgb(87,166,74)">ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);</span></div><div><span style="color:rgb(87,166,74)">ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);</span></div></div>
<div><span
style="color:rgb(87,166,74)"><br>
</span></div>
<div>Then, to
create the
several
matrices of
the levels of
TELESCOPE, I
thought using
the same
process as
before with
DMKSPSetComputeOperators.
But it doesn't
work...<br>
</div>
<div>Could you
tell me how I
can make it
work ?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Loic<br>
</div>
<div><br>
</div>
<div><span
style="color:rgb(87,166,74)"></span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their experiments
is infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>