<div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Tue 17. May 2022 at 11:56, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu">loic.gouarin@polytechnique.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Dave,</p>
<p>could you explain what you mean by state ?</p></div></blockquote><div dir="auto"><br></div><div dir="auto">Ah - by state Matt and I mean any auxiliary data stored in the user context passed to KSPSetComputeOperators which is required to define your operator and is distributed. For example, a Vec which is used store coefficients of your PDE.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><p></p>
<p>Thanks,</p>
<p>Loic<br>
</p></div><div text="#000000" bgcolor="#FFFFFF">
<div>Le 17/05/2022 à 11:50, Dave May a
écrit :<br>
</div>
<blockquote type="cite">
<div>
<div dir="auto">Hi Loic,</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Can you confirm if your problem has stored state
which needs to be repartitioned?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Dave</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Tue 17. May 2022 at 09:07, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div>Le 17/05/2022 à 01:43, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 17 May
2022 at 00:12, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, May 16, 2022 at 9:59 AM
Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Le 16/05/2022 à 21:54, Barry Smith a
écrit :<br>
</p>
<blockquote type="cite"> <br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at 3:50 PM,
Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Thanks Barry for your quick
reply.<br>
</p>
<div>Le 16/05/2022 à 21:41,
Barry Smith a écrit :<br>
</div>
<blockquote type="cite">
<div><br>
</div>
Loic,
<div><br>
</div>
<div> From your code it
looks like you are using a
DM. Is it a DMDA or a shell
DM? If it is a DMDA then the
process is intended to be
pretty straightforward.
PCTELESCOPE should create a
new DMDA that has the same
properties as the coarse
grid DMDA but lives on a
smaller number of ranks.
From this you can just
provide multiple levels of
coarsening of the DMDA to
produce the smaller
multigrid problems. <br>
</div>
</blockquote>
It's a DMDA. So what you mean is
to take the KSP of PCTELESCOPE,
get the DM which is the same of
the coarse grid and build all
the levels as previously for the
levels of PCTELESCOPE ?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
Yes, I think you should be able to
pull out of the coarse level PC (on
the subset of ranks) a DM that and
then setup coarse levels from that.
But you need to look at an example
that that uses telescope and probably
even the telescope code itself to see
all the details. I don't know them
anymore.</div>
</blockquote>
<p>I looked at the source code but It
didn't help. I will try to explain more
clearly my experiments to understand
what I make wrong and you can help more
easily.</p>
</div>
</blockquote>
<div>Hi Loic!</div>
<div><br>
</div>
<div>Here is my guess. When TELESCOPE
redistributes the coarse grid DMDA, it
creates a new DMDA and copies over all the
information it can see. However, I think you
may have state that is attached to the old
DMDA that is not getting redistributed. If
this is true, I can help you write a hook
that redistributes that state when TELESCOPE
maps between distributions.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you have state you need to propagate between
different communicators, you will have to use the
telescope type which is internally is referred to
as <br>
<pre width="80">TELESCOPE_COARSEDM</pre>
</div>
<div><br>
</div>
<div>To activate this type you need the option <br>
<pre width="80"><font color="#666666">-pc_telescope_use_coarse_dm</font></pre>
</div>
<div>Or to call PCTelescopeSetUseCoarseDM()</div>
<div>The assumptions of usng this options are
described here</div>
<div><a href="https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM" target="_blank">https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM</a></div>
<div><br>
</div>
<div>and source associated with the COARSEDM <br>
</div>
<div><a href="https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html" target="_blank">https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html</a></div>
<div><br>
</div>
<div>Using TELESCOPE_COARSEDM you will have access
to the hooks Matt is referring to.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<p>Thanks Matthew and Dave.</p>
<p>I used TELESCOPE_COARSEDM in my previous tests but it
tells me that "Zero instances of a coarse DM were
found". And indeed, my coarse solver only has the DM on
the coarse mesh and not on the fine level with a coarse
DM atttached. I think it's because I use
DMKSPSetComputeOperators to create the KSP on each
level.</p>
<p>So for now, I just added <br>
</p>
<div style="color:#d4d4d4;background-color:#1e1e1e;font-family:Menlo,Monaco,'Courier New',monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:#d4d4d4"> KSP coarse;</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCMGGetCoarseSolve</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_i</span><span style="color:#d4d4d4">, </span><span style="color:#b4b4b4">&</span><span style="color:#d4d4d4">coarse);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> PC </span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">;</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">KSPGetPC</span><span style="color:#d4d4d4">(coarse, </span><span style="color:#b4b4b4">&</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCSetType</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">, PCTELESCOPE);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div><div><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4"> </span><span style="color:#b4b4b4">=</span><span style="color:#d4d4d4"> </span><span style="color:#c8c8c8">PCTelescopeSetUseCoarseDM</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">pc_coarse</span><span style="color:#d4d4d4">, PETSC_TRUE);</span><span style="color:#c8c8c8">CHKERRQ</span><span style="color:#d4d4d4">(</span><span style="color:#c8c8c8">ierr</span><span style="color:#d4d4d4">);</span></div></div>
<p>but it's not enough.</p>
<p>Cheers,</p>
<p>Loic<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>Cheers,</div>
<div>Dave<br>
</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<div><br>
</div>
<div> Can you let us know
more details of what exactly
goes wrong with attempts? It
is likely straightforward to
get things working, perhaps
due to our unclear
documentation.</div>
</blockquote>
<p>I essentially have wrong
state or NULL matrices into
the PCTELESCOPE.</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at
3:28 PM, Loic Gouarin
<<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I could have
posted my message
on the user list
but it seems to me
that it's more in
the petsc
development part.
Don't hesitate to
tell me if I'm
wrong.<br>
I am developing a
code called cafes
that studies
fluid-particle
interactions in a
Stokes fluid. For
that, we
implemented the
whole solver in
matrix free on a
cartesian grid.
The preconditioner
is naturally a
geometrical
multigrid for the
velocity. The set
of matrix free for
each level is set
by hand (I worked
with Matthew on
this a long time
ago now!). <br>
I would have liked
to implement
telescope on this
part. I started it
but I didn't get
out of it. So I
would like to have
your help.<br>
What I tried to do
was to describe my
classical
multigrid up to a
level N1 and then
take the coarse
solver and apply
the telescope
preconditioner to
it up to a level
N2. I don't know
if this is the
best way but it
seemed the most
intuitive. The
problem is that I
could not get the
coarse matrix
because the setup
is not done yet (I
use a
DMKSPSetComputeOperators
for the matrices).
<br>
</p>
<p>The construction
of the matrix free
for each level is
here: <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88</a></p>
<p>The description
of the Stokes
preconditioner is
described here : <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162</a></p>
<p>I have tried
multiple
implementations
without success.
For me, the start
is to add at line
105 something like</p>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(87,166,74)">KSP coarse;</span></div><div><span style="color:rgb(87,166,74)">ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);</span></div><span style="color:rgb(87,166,74)">
</span></div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><span style="color:rgb(87,166,74)">PC pc_coarse;</span><div><span style="color:rgb(87,166,74)">ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);</span></div><div><span style="color:rgb(87,166,74)">ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);</span></div></div>
<div><span style="color:rgb(87,166,74)"><br>
</span></div>
<div>Then, to create
the several
matrices of the
levels of
TELESCOPE, I
thought using the
same process as
before with
DMKSPSetComputeOperators.
But it doesn't
work...<br>
</div>
<div>Could you tell
me how I can make
it work ?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Loic<br>
</div>
<div><br>
</div>
<div><span style="color:rgb(87,166,74)"></span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take
for granted before they begin
their experiments is infinitely
more interesting than any results
to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div>