<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 17 May 2022 at 00:12, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, May 16, 2022 at 9:59 AM Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Le 16/05/2022 à 21:54, Barry Smith a
      écrit :<br></p>
    <blockquote type="cite">
      
      <br>
      <div><br>
        <blockquote type="cite">
          <div>On May 16, 2022, at 3:50 PM, Loic Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
            wrote:</div>
          <br>
          <div>
            
            <div bgcolor="#FFFFFF">
              <p>Thanks Barry for your quick reply.<br>
              </p>
              <div>Le 16/05/2022 à 21:41, Barry
                Smith a écrit :<br>
              </div>
              <blockquote type="cite">
                
                <div><br>
                </div>
                  Loic,
                <div><br>
                </div>
                <div>    From your code it looks like you are
                  using a DM. Is it a DMDA or a shell DM? If it is a
                  DMDA then the process is intended to be pretty
                  straightforward. PCTELESCOPE should create a new DMDA
                  that has the same properties as the coarse grid DMDA
                  but lives on a smaller number of ranks. From this you
                  can just provide multiple levels of coarsening of the
                  DMDA to produce the smaller multigrid problems. <br>
                </div>
              </blockquote>
              It's a DMDA. So what you mean is to take the KSP of
              PCTELESCOPE, get the DM which is the same of the coarse
              grid and build all the levels as previously for the levels
              of PCTELESCOPE ?<br>
            </div>
          </div>
        </blockquote>
        <div><br>
        </div>
          Yes, I think you should be able to pull out of the coarse
        level PC (on the subset of ranks) a DM that and then setup
        coarse levels from that. But you need to look at an example that
        that uses telescope and probably even the telescope code itself
        to see all the details. I don't know them anymore.</div>
    </blockquote>
    <p>I looked at the source code but It didn't help. I will try to
      explain more clearly my experiments to understand what I make
      wrong and you can help more easily.</p>
    <p></p></div></blockquote><div>Hi Loic!</div><div><br></div><div>Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it creates a new DMDA and copies over all the information it can see. However, I think you may have state that is attached to the old DMDA that is not getting redistributed. If this is true, I can help you write a hook that redistributes that state when TELESCOPE maps between distributions.</div></div></div></blockquote><div><br></div><div>If you have state you need to propagate between different communicators, you will have to use the telescope type which is internally is referred to as <br><pre width="80">TELESCOPE_COARSEDM</pre></div><div><br></div><div>To activate this type you need the option <br><pre width="80"><font color="#666666">-pc_telescope_use_coarse_dm</font></pre></div><div>Or to call PCTelescopeSetUseCoarseDM()</div><div>The assumptions of usng this options are described here</div><div><a href="https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM">https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM</a></div><div><br></div><div>and source associated with the COARSEDM <br></div><div><a href="https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html">https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html</a></div><div><br></div><div>Using TELESCOPE_COARSEDM you will have access to the hooks Matt is referring to.</div><div><br></div><div>Cheers,</div><div>Dave<br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>   Thanks,</div><div><br></div><div>     Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>Thanks,</p>
    <p>Loic<br>
    </p>
    <blockquote type="cite">
      <div><br>
      </div>
      <div>  Barry</div>
      <div><br>
        <blockquote type="cite">
          <div>
            <div bgcolor="#FFFFFF">
              <blockquote type="cite">
                <div><br>
                </div>
                <div>   Can you let us know more details of
                  what exactly goes wrong with attempts? It is likely
                  straightforward to get things working, perhaps due to
                  our unclear documentation.</div>
              </blockquote>
              <p>I essentially have wrong state or NULL
                matrices into the PCTELESCOPE.</p>
              <p>Loic<br>
              </p>
              <blockquote type="cite">
                <div><br>
                </div>
                <div>  Barry</div>
                <div><br>
                  <div><br>
                    <blockquote type="cite">
                      <div>On May 16, 2022, at 3:28 PM, Loic
                        Gouarin <<a href="mailto:loic.gouarin@polytechnique.edu" target="_blank">loic.gouarin@polytechnique.edu</a>>
                        wrote:</div>
                      <br>
                      <div>
                        
                        <div bgcolor="#FFFFFF">
                          <p>Hello,</p>
                          <p>I could have posted my message on
                            the user list but it seems to me that it's
                            more in the petsc development part. Don't
                            hesitate to tell me if I'm wrong.<br>
                            I am developing a code called cafes that
                            studies fluid-particle interactions in a
                            Stokes fluid. For that, we implemented the
                            whole solver in matrix free on a cartesian
                            grid. The preconditioner is naturally a
                            geometrical multigrid for the velocity. The
                            set of matrix free for each level is set by
                            hand (I worked with Matthew on this a long
                            time ago now!). <br>
                            I would have liked to implement telescope on
                            this part. I started it but I didn't get out
                            of it. So I would like to have your help.<br>
                            What I tried to do was to describe my
                            classical multigrid up to a level N1 and
                            then take the coarse solver and apply the
                            telescope preconditioner to it up to a level
                            N2. I don't know if this is the best way but
                            it seemed the most intuitive. The problem is
                            that I could not get the coarse matrix
                            because the setup is not done yet (I use a
                            DMKSPSetComputeOperators for the matrices).
                            <br>
                          </p>
                          <p>The construction of the matrix
                            free for each level is here: <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88</a></p>
                          <p>The description of the Stokes
                            preconditioner is described here : <a href="https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162" target="_blank">https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162</a></p>
                          <p>I have tried multiple
                            implementations without success. For me, the
                            start is to add at line 105 something like</p>
                          <div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><div><span style="color:rgb(87,166,74)">KSP coarse;</span></div><div><span style="color:rgb(87,166,74)">ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr);</span></div><span style="color:rgb(87,166,74)">
</span></div>
                          <div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre-wrap"><span style="color:rgb(87,166,74)">PC pc_coarse;</span><div><span style="color:rgb(87,166,74)">ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr);</span></div><div><span style="color:rgb(87,166,74)">ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr);</span></div></div>
                          <div><span style="color:rgb(87,166,74)"><br>
                            </span></div>
                          <div>Then, to create the several
                            matrices of the levels of TELESCOPE, I
                            thought using the same process as before
                            with DMKSPSetComputeOperators. But it
                            doesn't work...<br>
                          </div>
                          <div>Could you tell me how I can make
                            it work ?</div>
                          <div><br>
                          </div>
                          <div>Thanks,</div>
                          <div>Loic<br>
                          </div>
                          <div><br>
                          </div>
                          <div><span style="color:rgb(87,166,74)"></span></div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </blockquote>
            </div>
          </div>
        </blockquote>
      </div>
      <br>
    </blockquote>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>