<div dir="ltr"><div dir="ltr">On Tue, May 17, 2022 at 4:53 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com">dave.mayhem23@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dear Loic,</div><div><br></div><div>I can confirm that PCTELESCOPE works nicely when using KSPSetComputeOperators. <br></div><div>Here is an example</div><div><br></div><div>petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2 <br></div><div><br></div><div>There is no special code in ex29.c to make the above options run.<br></div><div><br></div><div>I must also apologize for sending you down the garden path with COARSEDM. That really isn't what you want to use - I am sorry for the confusion on my part.</div><div>COARSEDM is intended for the case when YOU (or rather your coarse DM) define the sub-communicator, rather than having PCTELESCOPE define the sub-communicator.</div><div><br></div><div>I believe that PCTELESCOPE should works with DMKSPSetComputeOperators().</div><div><br></div><div>The function you provide to DMKSPSetComputeOperators() is unusual in my opinion.</div><div>Normally the method provided to DMKSPSetComputeOperators() just "assembles" (assembles meaning insert non-zero values for an AIJ or just calls MatAssemblyBegin/End for a matrix-free operator). <br></div><div>However your method seems to set sizes, set methods and call SetFromOptions. These are typically things which should done when the matrix is created - not when it is being "assembled".<br></div><div><br></div><div>Maybe this is part of the reason your code isn't playing nicely with telescope.</div><div>I think the code would be cleaner and if you overloaded your DMCreateMatrix() with a method which would return your matrix-free MATSHELL.</div><div><br></div><div>Yes I realize I am not directly helping solve your problem but maybe indirectly I am. Matt?</div></div></blockquote><div><br></div><div>Yes, I agree with Dave. This refactoring, even if it does not solve the problem, should allow us to understand it better.</div><div>Does the change make sense?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Cheers,</div><div>Dave<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 17 May 2022 at 13:18, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div>Le 17/05/2022 à 12:03, Dave May a
écrit :<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Tue 17. May 2022 at 11:56, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Dave,</p>
<p>could you explain what you mean by state ?</p>
</div>
</blockquote>
<div dir="auto"><br>
</div>
<div dir="auto">Ah - by state Matt and I mean any auxiliary
data stored in the user context passed to
KSPSetComputeOperators which is required to define your
operator and is distributed. For example, a Vec which is
used store coefficients of your PDE.</div>
<div dir="auto"><br>
</div>
</div>
</div>
</blockquote>
<p>Yes, I have a context which gives the operators (mult and
diagonal) and some other informations to build the matrix free on
each multigrid level.</p>
<p>You can find the idea here:
<a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91</a></p>
<p>It's the definition of the operator used inside
DMKSPComputeOperators.</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div>
<div class="gmail_quote">
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
</div>
<div bgcolor="#FFFFFF">
<div>Le 17/05/2022 à 11:50, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div>
<div dir="auto">Hi Loic,</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Can you confirm if your problem has
stored state which needs to be repartitioned?</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Dave</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Tue 17. May 2022 at 09:07, Loic
Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div>Le 17/05/2022 à 01:43, Dave May a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue,
17 May 2022 at 00:12, Matthew Knepley
<<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, May 16, 2022 at
9:59 AM Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Le 16/05/2022 à 21:54, Barry
Smith a écrit :<br>
</p>
<blockquote type="cite"> <br>
<div><br>
<blockquote type="cite">
<div>On May 16, 2022, at
3:50 PM, Loic Gouarin
<<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Thanks Barry for
your quick reply.<br>
</p>
<div>Le 16/05/2022 à
21:41, Barry Smith a
écrit :<br>
</div>
<blockquote type="cite">
<div><br>
</div>
Loic,
<div><br>
</div>
<div> From your
code it looks like
you are using a
DM. Is it a DMDA
or a shell DM? If
it is a DMDA then
the process is
intended to be
pretty
straightforward.
PCTELESCOPE should
create a new DMDA
that has the same
properties as the
coarse grid DMDA
but lives on a
smaller number of
ranks. From this
you can just
provide multiple
levels of
coarsening of the
DMDA to produce
the smaller
multigrid
problems. <br>
</div>
</blockquote>
It's a DMDA. So what
you mean is to take
the KSP of
PCTELESCOPE, get the
DM which is the same
of the coarse grid and
build all the levels
as previously for the
levels of PCTELESCOPE
?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
Yes, I think you should be
able to pull out of the
coarse level PC (on the
subset of ranks) a DM that
and then setup coarse levels
from that. But you need to
look at an example that that
uses telescope and probably
even the telescope code
itself to see all the
details. I don't know them
anymore.</div>
</blockquote>
<p>I looked at the source code
but It didn't help. I will try
to explain more clearly my
experiments to understand what
I make wrong and you can help
more easily.</p>
</div>
</blockquote>
<div>Hi Loic!</div>
<div><br>
</div>
<div>Here is my guess. When
TELESCOPE redistributes the coarse
grid DMDA, it creates a new DMDA
and copies over all the
information it can see. However, I
think you may have state that is
attached to the old DMDA that is
not getting redistributed. If this
is true, I can help you write a
hook that redistributes that state
when TELESCOPE maps between
distributions.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you have state you need to
propagate between different
communicators, you will have to use the
telescope type which is internally is
referred to as <br>
<pre width="80">TELESCOPE_COARSEDM</pre>
</div>
<div><br>
</div>
<div>To activate this type you need the
option <br>
<pre width="80"><font color="#666666">-pc_telescope_use_coarse_dm</font></pre>
</div>
<div>Or to call
PCTelescopeSetUseCoarseDM()</div>
<div>The assumptions of usng this options
are described here</div>
<div><a href="https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM" target="_blank">https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM</a></div>
<div><br>
</div>
<div>and source associated with the
COARSEDM <br>
</div>
<div><a href="https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html" target="_blank">https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html</a></div>
<div><br>
</div>
<div>Using TELESCOPE_COARSEDM you will
have access to the hooks Matt is
referring to.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<p>Thanks Matthew and Dave.</p>
<p>I used TELESCOPE_COARSEDM in my previous
tests but it tells me that "Zero instances of
a coarse DM were found". And indeed, my coarse
solver only has the DM on the coarse mesh and
not on the fine level with a coarse DM
atttached. I think it's because I use
DMKSPSetComputeOperators to create the KSP on
each level.</p>
<p>So for now, I just added <br>
</p>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(212,212,212)"> KSP coarse;</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(180,180,180)">=</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">PCMGGetCoarseSolve</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">pc_i</span><span style="color:rgb(212,212,212)">, </span><span style="color:rgb(180,180,180)">&</span><span style="color:rgb(212,212,212)">coarse);</span><span style="color:rgb(200,200,200)">CHKERRQ</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)">);</span></div><div><span style="color:rgb(212,212,212)"> PC </span><span style="color:rgb(200,200,200)">pc_coarse</span><span style="color:rgb(212,212,212)">;</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(180,180,180)">=</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">KSPGetPC</span><span style="color:rgb(212,212,212)">(coarse, </span><span style="color:rgb(180,180,180)">&</span><span style="color:rgb(200,200,200)">pc_coarse</span><span style="color:rgb(212,212,212)">);</span><span style="color:rgb(200,200,200)">CHKERRQ</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)">);</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(180,180,180)">=</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">PCSetType</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">pc_coarse</span><span style="color:rgb(212,212,212)">, PCTELESCOPE);</span><span style="color:rgb(200,200,200)">CHKERRQ</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)">);</span></div><div><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(180,180,180)">=</span><span style="color:rgb(212,212,212)"> </span><span style="color:rgb(200,200,200)">PCTelescopeSetUseCoarseDM</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">pc_coarse</span><span style="color:rgb(212,212,212)">, PETSC_TRUE);</span><span style="color:rgb(200,200,200)">CHKERRQ</span><span style="color:rgb(212,212,212)">(</span><span style="color:rgb(200,200,200)">ierr</span><span style="color:rgb(212,212,212)">);</span></div></div>
<p>but it's not enough.</p>
<p>Cheers,</p>
<p>Loic<br>
</p>
</div>
<div bgcolor="#FFFFFF">
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>Cheers,</div>
<div>Dave<br>
</div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<blockquote type="cite">
<div>
<div bgcolor="#FFFFFF">
<blockquote type="cite">
<div><br>
</div>
<div> Can you let
us know more
details of what
exactly goes wrong
with attempts? It
is likely
straightforward to
get things
working, perhaps
due to our unclear
documentation.</div>
</blockquote>
<p>I essentially have
wrong state or NULL
matrices into the
PCTELESCOPE.</p>
<p>Loic<br>
</p>
<blockquote type="cite">
<div><br>
</div>
<div> Barry</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On May
16, 2022, at
3:28 PM, Loic
Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I could
have posted my
message on the
user list but
it seems to me
that it's more
in the petsc
development
part. Don't
hesitate to
tell me if I'm
wrong.<br>
I am
developing a
code called
cafes that
studies
fluid-particle
interactions
in a Stokes
fluid. For
that, we
implemented
the whole
solver in
matrix free on
a cartesian
grid. The
preconditioner
is naturally a
geometrical
multigrid for
the velocity.
The set of
matrix free
for each level
is set by hand
(I worked with
Matthew on
this a long
time ago
now!). <br>
I would have
liked to
implement
telescope on
this part. I
started it but
I didn't get
out of it. So
I would like
to have your
help.<br>
What I tried
to do was to
describe my
classical
multigrid up
to a level N1
and then take
the coarse
solver and
apply the
telescope
preconditioner
to it up to a
level N2. I
don't know if
this is the
best way but
it seemed the
most
intuitive. The
problem is
that I could
not get the
coarse matrix
because the
setup is not
done yet (I
use a
DMKSPSetComputeOperators
for the
matrices). <br>
</p>
<p>The
construction
of the matrix
free for each
level is here:
<a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88</a></p>
<p>The
description of
the Stokes
preconditioner
is described
here : <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162</a></p>
<p>I have
tried multiple
implementations without success. For me, the start is to add at line 105
something like</p>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(87,166,74)">KSP coarse;</span></div><div><span style="color:rgb(87,166,74)">ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);</span></div><span style="color:rgb(87,166,74)">
</span></div>
<div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><span style="color:rgb(87,166,74)">PC pc_coarse;</span><div><span style="color:rgb(87,166,74)">ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);</span></div><div><span style="color:rgb(87,166,74)">ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);</span></div></div>
<div><span style="color:rgb(87,166,74)"><br>
</span></div>
<div>Then, to
create the
several
matrices of
the levels of
TELESCOPE, I
thought using
the same
process as
before with
DMKSPSetComputeOperators.
But it doesn't
work...<br>
</div>
<div>Could you
tell me how I
can make it
work ?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Loic<br>
</div>
<div><br>
</div>
<div><span style="color:rgb(87,166,74)"></span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their experiments
is infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>