<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">Le 17/05/2022 à 01:43, Dave May a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAJ98EDrK_zs0_qsHLMw7b5citG=VPL3n+n7P8-qxhpc=AMrhaw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 17 May 2022 at
00:12, Matthew Knepley <<a
href="mailto:knepley@gmail.com" moz-do-not-send="true"
class="moz-txt-link-freetext">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, May 16, 2022 at 9:59 AM Loic
Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Le 16/05/2022 à 21:54, Barry Smith a écrit :<br>
</p>
<blockquote type="cite"> <br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at 3:50 PM, Loic Gouarin
<<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Thanks Barry for your quick reply.<br>
</p>
<div>Le 16/05/2022 à 21:41, Barry Smith a
écrit :<br>
</div>
<blockquote type="cite">
<div><br>
</div>
Loic,
<div><br>
</div>
<div> From your code it looks like
you are using a DM. Is it a DMDA or a
shell DM? If it is a DMDA then the
process is intended to be pretty
straightforward. PCTELESCOPE should
create a new DMDA that has the same
properties as the coarse grid DMDA but
lives on a smaller number of ranks.
From this you can just provide
multiple levels of coarsening of the
DMDA to produce the smaller multigrid
problems. <br>
</div>
</blockquote>
It's a DMDA. So what you mean is to take
the KSP of PCTELESCOPE, get the DM which
is the same of the coarse grid and build
all the levels as previously for the
levels of PCTELESCOPE ?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
Yes, I think you should be able to pull out of
the coarse level PC (on the subset of ranks) a
DM that and then setup coarse levels from that.
But you need to look at an example that that
uses telescope and probably even the telescope
code itself to see all the details. I don't know
them anymore.</div>
</blockquote>
<p>I looked at the source code but It didn't help. I
will try to explain more clearly my experiments to
understand what I make wrong and you can help more
easily.</p>
</div>
</blockquote>
<div>Hi Loic!</div>
<div><br>
</div>
<div>Here is my guess. When TELESCOPE redistributes the
coarse grid DMDA, it creates a new DMDA and copies
over all the information it can see. However, I think
you may have state that is attached to the old DMDA
that is not getting redistributed. If this is true, I
can help you write a hook that redistributes that
state when TELESCOPE maps between distributions.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you have state you need to propagate between different
communicators, you will have to use the telescope type which
is internally is referred to as <br>
<pre width="80">TELESCOPE_COARSEDM</pre>
</div>
<div><br>
</div>
<div>To activate this type you need the option <br>
<pre width="80"><font color="#666666">-pc_telescope_use_coarse_dm</font></pre>
</div>
<div>Or to call PCTelescopeSetUseCoarseDM()</div>
<div>The assumptions of usng this options are described here</div>
<div><a
href="https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM"
moz-do-not-send="true" class="moz-txt-link-freetext">https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM</a></div>
<div><br>
</div>
<div>and source associated with the COARSEDM <br>
</div>
<div><a
href="https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html"
moz-do-not-send="true" class="moz-txt-link-freetext">https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html</a></div>
<div><br>
</div>
<div>Using TELESCOPE_COARSEDM you will have access to the
hooks Matt is referring to.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<p>Thanks Matthew and Dave.</p>
<p>I used TELESCOPE_COARSEDM in my previous tests but it tells me
that "Zero instances of a coarse DM were found". And indeed, my
coarse solver only has the DM on the coarse mesh and not on the
fine level with a coarse DM atttached. I think it's because I use
DMKSPSetComputeOperators to create the KSP on each level.</p>
<p>So for now, I just added <br>
</p>
<div style="color: #d4d4d4;background-color: #1e1e1e;font-family: Menlo, Monaco, 'Courier New', monospace;font-weight: normal;font-size: 12px;line-height: 18px;white-space: pre;"><div><span style="color: #d4d4d4;"> KSP coarse;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;"> </span><span style="color: #b4b4b4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">PCMGGetCoarseSolve</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">pc_i</span><span style="color: #d4d4d4;">, </span><span style="color: #b4b4b4;">&</span><span style="color: #d4d4d4;">coarse);</span><span style="color: #c8c8c8;">CHKERRQ</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> PC </span><span style="color: #c8c8c8;">pc_coarse</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;"> </span><span style="color: #b4b4b4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">KSPGetPC</span><span style="color: #d4d4d4;">(coarse, </span><span style="color: #b4b4b4;">&</span><span style="color: #c8c8c8;">pc_coarse</span><span style="color: #d4d4d4;">);</span><span style="color: #c8c8c8;">CHKERRQ</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;"> </span><span style="color: #b4b4b4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">PCSetType</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">pc_coarse</span><span style="color: #d4d4d4;">, PCTELESCOPE);</span><span style="color: #c8c8c8;">CHKERRQ</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;"> </span><span style="color: #b4b4b4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #c8c8c8;">PCTelescopeSetUseCoarseDM</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">pc_coarse</span><span style="color: #d4d4d4;">, PETSC_TRUE);</span><span style="color: #c8c8c8;">CHKERRQ</span><span style="color: #d4d4d4;">(</span><span style="color: #c8c8c8;">ierr</span><span style="color: #d4d4d4;">);</span></div></div>
<p>but it's not enough.</p>
<p>Cheers,</p>
<p>Loic<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAJ98EDrK_zs0_qsHLMw7b5citG=VPL3n+n7P8-qxhpc=AMrhaw@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div>Cheers,</div>
<div>Dave<br>
</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<div><br>
</div>
<div> Can you let us know more details
of what exactly goes wrong with
attempts? It is likely straightforward
to get things working, perhaps due to
our unclear documentation.</div>
</blockquote>
<p>I essentially have wrong state or NULL
matrices into the PCTELESCOPE.</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at 3:28 PM,
Loic Gouarin <<a
href="mailto:loic.gouarin@polytechnique.edu"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I could have posted my
message on the user list but
it seems to me that it's
more in the petsc
development part. Don't
hesitate to tell me if I'm
wrong.<br>
I am developing a code
called cafes that studies
fluid-particle interactions
in a Stokes fluid. For that,
we implemented the whole
solver in matrix free on a
cartesian grid. The
preconditioner is naturally
a geometrical multigrid for
the velocity. The set of
matrix free for each level
is set by hand (I worked
with Matthew on this a long
time ago now!). <br>
I would have liked to
implement telescope on this
part. I started it but I
didn't get out of it. So I
would like to have your
help.<br>
What I tried to do was to
describe my classical
multigrid up to a level N1
and then take the coarse
solver and apply the
telescope preconditioner to
it up to a level N2. I don't
know if this is the best way
but it seemed the most
intuitive. The problem is
that I could not get the
coarse matrix because the
setup is not done yet (I use
a DMKSPSetComputeOperators
for the matrices). <br>
</p>
<p>The construction of the
matrix free for each level
is here: <a
href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88</a></p>
<p>The description of the
Stokes preconditioner is
described here : <a
href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162</a></p>
<p>I have tried multiple
implementations without
success. For me, the start
is to add at line 105
something like</p>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(87,166,74)">KSP coarse;</span></div><div><span style="color:rgb(87,166,74)">ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);</span></div><span style="color:rgb(87,166,74)">
</span></div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><span style="color:rgb(87,166,74)">PC pc_coarse;</span><div><span style="color:rgb(87,166,74)">ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);</span></div><div><span style="color:rgb(87,166,74)">ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);</span></div></div>
<div><span
style="color:rgb(87,166,74)"><br>
</span></div>
<div>Then, to create the
several matrices of the
levels of TELESCOPE, I
thought using the same
process as before with
DMKSPSetComputeOperators.
But it doesn't work...<br>
</div>
<div>Could you tell me how I
can make it work ?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Loic<br>
</div>
<div><br>
</div>
<div><span
style="color:rgb(87,166,74)"></span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted
before they begin their experiments is
infinitely more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>