<html>
<head></head>
<body>
<div style="font-family:sans-serif"> <span dir="ltr" style="margin-top:0; margin-bottom:0;">Thanks Dave for your time and this detailed reply.</span>
<br>
<br> <span dir="ltr" style="margin-top:0; margin-bottom:0;">My implementation is old and should probably be updated.</span>
<br>
<br> <span dir="ltr" style="margin-top:0; margin-bottom:0;">I will read your email carefully and I come back to you with a better implementation.</span>
<br>
<br> <span dir="ltr" style="margin-top:0; margin-bottom:0;">Thanks,</span>
<br> <span dir="ltr" style="margin-top:0; margin-bottom:0;">Loic</span>
<br>
</div>
<div>
<br>
<div>
<p>17 mai 2022 16:53:37 Dave May <dave.mayhem23@gmail.com>:</p>
</div>
<blockquote style="margin:0;border-left:3px solid #ccc; padding-left:10px">
<div dir="ltr">
<div>
Dear Loic,
</div>
<div>
<br>
</div>
<div>
I can confirm that PCTELESCOPE works nicely when using KSPSetComputeOperators.
<br>
</div>
<div>
Here is an example
</div>
<div>
<br>
</div>
<div>
petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2
<br>
</div>
<div>
<br>
</div>
<div>
There is no special code in ex29.c to make the above options run.
<br>
</div>
<div>
<br>
</div>
<div>
I must also apologize for sending you down the garden path with COARSEDM. That really isn't what you want to use - I am sorry for the confusion on my part.
</div>
<div>
COARSEDM is intended for the case when YOU (or rather your coarse DM) define the sub-communicator, rather than having PCTELESCOPE define the sub-communicator.
</div>
<div>
<br>
</div>
<div>
I believe that PCTELESCOPE should works with DMKSPSetComputeOperators().
</div>
<div>
<br>
</div>
<div>
The function you provide to DMKSPSetComputeOperators() is unusual in my opinion.
</div>
<div>
Normally the method provided to DMKSPSetComputeOperators() just "assembles" (assembles meaning insert non-zero values for an AIJ or just calls MatAssemblyBegin/End for a matrix-free operator).
<br>
</div>
<div>
However your method seems to set sizes, set methods and call SetFromOptions. These are typically things which should done when the matrix is created - not when it is being "assembled".
<br>
</div>
<div>
<br>
</div>
<div>
Maybe this is part of the reason your code isn't playing nicely with telescope.
</div>
<div>
I think the code would be cleaner and if you overloaded your DMCreateMatrix() with a method which would return your matrix-free MATSHELL.
</div>
<div>
<br>
</div>
<div>
Yes I realize I am not directly helping solve your problem but maybe indirectly I am. Matt?
<br>
</div>
<div>
<br>
</div>
<div>
Cheers,
</div>
<div>
Dave
<br>
</div>
<div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">
On Tue, 17 May 2022 at 13:18, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu">loic.gouarin@polytechnique.edu</a>> wrote:
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br> </p>
<div>
Le 17/05/2022 à 12:03, Dave May a écrit :
<br>
</div>
<blockquote type="cite">
<div>
<br>
</div>
<div>
<br>
<div class="gmail_quote">
<div dir="ltr">
On Tue 17. May 2022 at 11:56, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>> wrote:
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Dave,</p>
<p>could you explain what you mean by state ?</p>
</div>
</blockquote>
<div dir="auto">
<br>
</div>
<div dir="auto">
Ah - by state Matt and I mean any auxiliary data stored in the user context passed to KSPSetComputeOperators which is required to define your operator and is distributed. For example, a Vec which is used store coefficients of your PDE.
</div>
<div dir="auto">
<br>
</div>
</div>
</div>
</blockquote>
<p>Yes, I have a context which gives the operators (mult and diagonal) and some other informations to build the matrix free on each multigrid level.</p>
<p>You can find the idea here: <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91</a></p>
<p>It's the definition of the operator used inside DMKSPComputeOperators.</p>
<p>Loic<br> </p>
<blockquote type="cite">
<div>
<div class="gmail_quote">
<div dir="auto">
<br>
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thanks,</p>
<p>Loic<br> </p>
</div>
<div bgcolor="#FFFFFF">
<div>
Le 17/05/2022 à 11:50, Dave May a écrit :
<br>
</div>
<blockquote type="cite">
<div>
<div dir="auto">
Hi Loic,
</div>
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
Can you confirm if your problem has stored state which needs to be repartitioned?
</div>
<div dir="auto">
<br>
</div>
<div dir="auto">
Thanks,
</div>
<div dir="auto">
Dave
</div>
<div>
<br>
<div class="gmail_quote">
<div dir="ltr">
On Tue 17. May 2022 at 09:07, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>> wrote:
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
…
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</body>
</html>