<div dir="ltr"><div dir="ltr">On Thu, Apr 14, 2022 at 8:30 AM Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com">chenzhuotj@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Pierre,<div><br></div><div>Thank you so much! It helps a lot! Petsc correctly identifies the neighboring cells and the geometry is quite complicated as shown below.</div><img src="cid:ii_l1yz70fw0" alt="dd.png" width="562" height="487"><div><br></div><div>I guess Petsc identifies the non-zero off-diagonal rows and columns and marks them as the 1 layer overlap region. For more layers, one needs the off-diagonal information of A*A (I am solving Ax=b), is that correct?</div></div></blockquote><div><br></div><div>For more layers, you are just doing Breadth First Search in the matrix graph.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Best regards.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 14, 2022 at 6:44 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On 14 Apr 2022, at 12:16 PM, Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Hi Matt and Pierre,<div><br></div><div>Thank you very much for your help. I indeed want to solve a 2D problem with geometric block domain decomposition and the blocks are in the Hilbert space curve ordering. Each block may have 32x32 cells.<br><div><br></div><div>I am a Fortran user and I apologize that I do not know how to use -pc_asm_print_subdomains.</div></div></div></div></blockquote><div><br></div><div>You can just put this on your command line. For example,</div><div>$ cd ${PETSC_DIR}/src/ksp/ksp/tutorials && make ex2f && mpirun -n 2 ./ex2f -pc_type asm -pc_asm_print_subdomains</div>[…]</div><div>[0:2] Subdomain 0 with overlap:<br>0 1 2 3 4 <br>[1:2] Subdomain 0 with overlap:<br>5 6 7 8 <br>Norm of error 0.1205E-04 iterations 4<br><br><blockquote type="cite"><div><div dir="ltr"><div><div> Should I use call PetscOptionsGetBool(NULL, NULL, "-pc_asm_print_subdomains", .true., NULL, ierr)? It would be great if you can point me to the right tutorial.</div></div></div></div></blockquote><div><br></div><div>You could also use the following line of code:</div><div> call PetscOptionsInsertString(PETSC_NULL_OPTIONS,'-pc_asm_print_subdomains',ierr)</div><br><blockquote type="cite"><div><div dir="ltr"><div><div>After reading some example programs, I would like to know if it is possible to use</div><div><br></div><div>IS, allocatable, dimension(:) :: is1,is2<br></div><div>integer :: nlocalblk</div><div><br></div><div>call PCASMGetLocalSubdomains(pc,nlocalblk,is1,is2,ierr)</div><div>call ISView(is1,PETSC_VIEWER_STDOUT_SELF,ierr)</div><div>call ISView(is2,PETSC_VIEWER_STDOUT_SELF,ierr)</div><div><br></div><div>to realize the function you have mentioned.</div></div></div></div></blockquote><div><br></div><div>That is another way to do it, yes.</div><div>In fact, for debugging purposes, I’d advise that you use GASM instead of ASM because it prints slightly more coherent information.</div><div>E.g., with PCASM, the above log shows that the displayed information is _wrong_. These are not the subdomains with overlap, but the subdomains _without_ overlap.</div><div>Furthermore, if you increase -pc_asm_overlap, you’ll still get the same subdomains.</div><div>$ mpirun -n 2 ./ex2f -pc_type asm -pc_asm_overlap 4 -pc_asm_print_subdomains </div>[0:2] Subdomain 0 with overlap:<br>0 1 2 3 4 <br>[1:2] Subdomain 0 with overlap:<br>5 6 7 8 <br>Norm of error 0.1192E-05 iterations 4<br><div><br></div><div>On the contrary, with PCGASM, you get the proper subdomains with and without overlap.</div><div>$ mpirun -n 2 ./ex2f -pc_type gasm -pc_gasm_print_subdomains</div>Inner subdomain:<br>0 1 2 3 4 <br>Outer subdomain:<br>0 1 2 3 4 <br>Inner subdomain:<br>5 6 7 8 <br>Outer subdomain:<br>5 6 7 8 <br><div>$ mpirun -n 2 ./ex2f -pc_type gasm -pc_gasm_print_subdomains -pc_gasm_overlap 4</div>Inner subdomain:<br>0 1 2 3 4 <br>Outer subdomain:<br>0 1 2 3 4 5 6 7 8 <br>Inner subdomain:<br>5 6 7 8 <br>Outer subdomain:<br>0 1 2 3 4 5 6 7 8 <br><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div><div>Thank you very much!<br><div><br></div><div><br></div><div><br></div></div></div></div><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 10:18 PM Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you, Pierre!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 10:05 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>You can also use the uncommented option -pc_asm_print_subdomains which will, as Matt told you, show you that it is exactly the same algorithm.<div><br></div><div>Thanks,</div><div>Pierre<br><div><br><blockquote type="cite"><div>On 13 Apr 2022, at 3:58 PM, Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Thank you, Matt! I will do that.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 9:55 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Apr 13, 2022 at 9:53 AM Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear Pierre,<div><br></div><div>Thank you! I looked into the webpage you sent me and I think it is not the situation that I am talking about.</div><div><br></div><div>I think I need to attach a figure for an illustrative purpose. This figure is Figure 14.5 of "Iterative Method for Sparse Linear Systems" by Saad.</div><div><span id="gmail-m_-1668360613991711737gmail-m_644854792250835976gmail-m_-7875060808135279206gmail-m_3377797180640090925gmail-m_3141105389219103399cid:ii_l1xmo4fd0"><domaindecompostion.png></span></div><div><br></div><div>If I divide the domain into these three subdomains, as you can see, the middle block has two interfaces. In the matrix form, its rows are not contiguous, i.e., distributed in different processors. If ASM only expands in the contiguous direction, the domain decomposition become ineffective, I guess.</div></div></blockquote><div><br></div><div>No, we get exactly this picture. Saad is talking about exactly the algorithm we use.</div><div><br></div><div>Maybe you should just look at the subdomains being produced, -mat_view draw -draw_pause 3</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 9:36 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" target="_blank">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On 13 Apr 2022, at 3:30 PM, Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Dear Matthew and Mark,<div><br></div><div>Thank you very much for the reply! Much appreciated!</div><div><br></div><div>The question was about a 1D problem. I think I should say core 1 has row 1:32 instead of 1:32, 1:32 as it might be confusing.</div><div><br></div><div>So the overlap is extended to both directions for the middle processor but only toward the increasing direction for the first processor and the decreasing direction for the last processor. In 1D, this makes sense as the domain is contiguous. However, in 2D with domain decomposition with spacial overlaps, this overlapping would not work as one subdomain can have several neighbor domains. Mark mentioned generalized ASM, is that the correct direction that I should look for?</div></div></div></blockquote><div><br></div><div>What is it that you want to do exactly?</div><div>If you are using a standard discretisation kernel, e.g., piecewise linear finite elements, MatIncreaseOverlap() called by PCASM will generate an overlap algebraically which is equivalent to the overlap you would have gotten geometrically.</div><div>If you know that “geometric” overlap (or want to use a custom definition of overlap), you could use <a href="https://petsc.org/release/docs/manualpages/PC/PCASMSetLocalSubdomains.html" target="_blank">https://petsc.org/release/docs/manualpages/PC/PCASMSetLocalSubdomains.html</a></div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div>Best regards.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 9:14 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Apr 13, 2022 at 9:11 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 8:56 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Apr 13, 2022 at 6:42 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">No, without overlap you have, let say: <div><div>core 1: 1:32, 1:32</div><div>core 2: 33:64, 33:64</div></div><div><br></div><div>Overlap will increase the size of each domain so you get:</div><div><div>core 1: 1:33, 1:33</div><div>core 2: 32:65, 32:65</div></div></div></blockquote><div><br></div><div>I do not think this is correct. Here is the algorithm. Imagine the matrix is a large graph. When you divide rows, you</div><div>can think of that as dividing the vertices into sets. If overlap = 1, it means start with my vertex set, and add all vertices</div><div>that are just 1 edge away from my set.</div></div></div></blockquote><div><br></div><div>I think that is what was said. You increase each subdomain by one row of vertices.</div><div>So in 1D, vertex 32 and 33 are in both subdomains and you have an overlap region of size 2. </div><div>They want an overlap region of size 1, vertex 33.</div></div></div></blockquote><div><br></div><div>This is true, but I did not think they specified a 1D mesh.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>What you want is reasonable but requires PETSc to pick a separator set, which is not well defined.</div><div>You need to build that yourself with gasm (I think) if you want this.</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 3:17 AM Zhuo Chen <<a href="mailto:chenzhuotj@gmail.com" target="_blank">chenzhuotj@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I hope that everything is going well with everybody.</div><div><br></div><div>I have a question about the PCASMSetOverlap. If I have a 128x128 matrix and I use 4 cores with overlap=1. Does it mean that from core 1 to core 4, the block ranges are (starting from 1):</div><div><br></div><div>core 1: 1:33, 1:33</div><div>core 2: 33:65, 33:65</div><div>core 3: 65:97, 65:97</div><div>core 4: 95:128, 95:128</div><div><br></div><div>Or is it something else? I cannot tell from the manual.</div><div><br></div><div>Many thanks in advance.</div><div><br></div><div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font>Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font style="font-family:arial,sans-serif;font-size:12.8px" color="#000000">Zhuo Chen</font><div style="font-family:arial,sans-serif;font-size:12.8px"><font color="#000000">Department of Astronomy</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font color="#000000">Tsinghua University</font></div><div style="font-family:arial,sans-serif;font-size:12.8px"><font color="#000000">Beijing, China 100084</font></div><div><font face="'comic sans ms', sans-serif"><font color="#000000"><u><a href="https://czlovemath123.github.io/" target="_blank">https://czlovemath123.github.io/</a></u></font><br></font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>