[petsc-users] What does PCASMSetOverlap do?

Zhuo Chen chenzhuotj at gmail.com
Thu Apr 14 08:39:18 CDT 2022 

It is great to know that! Thank you, Matt.

On Thu, Apr 14, 2022 at 8:56 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Apr 14, 2022 at 8:30 AM Zhuo Chen <chenzhuotj at gmail.com> wrote:
>
>> Hi Pierre,
>>
>> Thank you so much! It helps a lot! Petsc correctly identifies the
>> neighboring cells and the geometry is quite complicated as shown below.
>> [image: dd.png]
>>
>> I guess Petsc identifies the non-zero off-diagonal rows and columns and
>> marks them as the 1 layer overlap region. For more layers, one needs the
>> off-diagonal information of A*A (I am solving Ax=b), is that correct?
>>
>
> For more layers, you are just doing Breadth First Search in the matrix
> graph.
>
>   Thanks,
>
>      Matt
>
>
>> Best regards.
>>
>>
>> On Thu, Apr 14, 2022 at 6:44 PM Pierre Jolivet <pierre at joliv.et> wrote:
>>
>>>
>>>
>>> On 14 Apr 2022, at 12:16 PM, Zhuo Chen <chenzhuotj at gmail.com> wrote:
>>>
>>> Hi Matt and Pierre,
>>>
>>> Thank you very much for your help. I indeed want to solve a 2D problem
>>> with geometric block domain decomposition and the blocks are in the Hilbert
>>> space curve ordering. Each block may have 32x32 cells.
>>>
>>> I am a Fortran user and I apologize that I do not know how to use
>>> -pc_asm_print_subdomains.
>>>
>>>
>>> You can just put this on your command line. For example,
>>> $ cd ${PETSC_DIR}/src/ksp/ksp/tutorials && make ex2f && mpirun -n 2
>>> ./ex2f -pc_type asm -pc_asm_print_subdomains
>>> […]
>>> [0:2] Subdomain 0 with overlap:
>>> 0 1 2 3 4
>>> [1:2] Subdomain 0 with overlap:
>>> 5 6 7 8
>>> Norm of error  0.1205E-04 iterations     4
>>>
>>> Should I use call PetscOptionsGetBool(NULL, NULL,
>>> "-pc_asm_print_subdomains", .true., NULL, ierr)? It would be great if you
>>> can point me to the right tutorial.
>>>
>>>
>>> You could also use the following line of code:
>>>       call
>>> PetscOptionsInsertString(PETSC_NULL_OPTIONS,'-pc_asm_print_subdomains',ierr)
>>>
>>> After reading some example programs, I would like to know if it is
>>> possible to use
>>>
>>> IS, allocatable, dimension(:) :: is1,is2
>>> integer :: nlocalblk
>>>
>>> call PCASMGetLocalSubdomains(pc,nlocalblk,is1,is2,ierr)
>>> call ISView(is1,PETSC_VIEWER_STDOUT_SELF,ierr)
>>> call ISView(is2,PETSC_VIEWER_STDOUT_SELF,ierr)
>>>
>>> to realize the function you have mentioned.
>>>
>>>
>>> That is another way to do it, yes.
>>> In fact, for debugging purposes, I’d advise that you use GASM instead of
>>> ASM because it prints slightly more coherent information.
>>> E.g., with PCASM, the above log shows that the displayed information is
>>> _wrong_. These are not the subdomains with overlap, but the subdomains
>>> _without_ overlap.
>>> Furthermore, if you increase -pc_asm_overlap, you’ll still get the same
>>> subdomains.
>>> $ mpirun -n 2 ./ex2f -pc_type asm -pc_asm_overlap 4
>>> -pc_asm_print_subdomains
>>> [0:2] Subdomain 0 with overlap:
>>> 0 1 2 3 4
>>> [1:2] Subdomain 0 with overlap:
>>> 5 6 7 8
>>> Norm of error  0.1192E-05 iterations     4
>>>
>>> On the contrary, with PCGASM, you get the proper subdomains with and
>>> without overlap.
>>> $ mpirun -n 2 ./ex2f -pc_type gasm -pc_gasm_print_subdomains
>>> Inner subdomain:
>>> 0 1 2 3 4
>>> Outer subdomain:
>>> 0 1 2 3 4
>>> Inner subdomain:
>>> 5 6 7 8
>>> Outer subdomain:
>>> 5 6 7 8
>>> $ mpirun -n 2 ./ex2f -pc_type gasm -pc_gasm_print_subdomains
>>> -pc_gasm_overlap 4
>>> Inner subdomain:
>>> 0 1 2 3 4
>>> Outer subdomain:
>>> 0 1 2 3 4 5 6 7 8
>>> Inner subdomain:
>>> 5 6 7 8
>>> Outer subdomain:
>>> 0 1 2 3 4 5 6 7 8
>>>
>>> Thanks,
>>> Pierre
>>>
>>> Thank you very much!
>>>
>>>
>>>
>>>
>>> On Wed, Apr 13, 2022 at 10:18 PM Zhuo Chen <chenzhuotj at gmail.com> wrote:
>>>
>>>> Thank you, Pierre!
>>>>
>>>> On Wed, Apr 13, 2022 at 10:05 PM Pierre Jolivet <pierre at joliv.et>
>>>> wrote:
>>>>
>>>>> You can also use the uncommented option -pc_asm_print_subdomains which
>>>>> will, as Matt told you, show you that it is exactly the same algorithm.
>>>>>
>>>>> Thanks,
>>>>> Pierre
>>>>>
>>>>> On 13 Apr 2022, at 3:58 PM, Zhuo Chen <chenzhuotj at gmail.com> wrote:
>>>>>
>>>>> Thank you, Matt! I will do that.
>>>>>
>>>>> On Wed, Apr 13, 2022 at 9:55 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Apr 13, 2022 at 9:53 AM Zhuo Chen <chenzhuotj at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Dear Pierre,
>>>>>>>
>>>>>>> Thank you! I looked into the webpage you sent me and I think it is
>>>>>>> not the situation that I am talking about.
>>>>>>>
>>>>>>> I think I need to attach a figure for an illustrative purpose. This
>>>>>>> figure is Figure 14.5 of "Iterative Method for Sparse Linear Systems" by
>>>>>>> Saad.
>>>>>>> <domaindecompostion.png>
>>>>>>>
>>>>>>> If I divide the domain into these three subdomains, as you can see,
>>>>>>> the middle block has two interfaces. In the matrix form, its rows are not
>>>>>>> contiguous, i.e., distributed in different processors. If ASM only expands
>>>>>>> in the contiguous direction, the domain decomposition become ineffective, I
>>>>>>> guess.
>>>>>>>
>>>>>>
>>>>>> No, we get exactly this picture. Saad is talking about exactly the
>>>>>> algorithm we use.
>>>>>>
>>>>>> Maybe you should just look at the subdomains being produced,
>>>>>> -mat_view draw -draw_pause 3
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>> On Wed, Apr 13, 2022 at 9:36 PM Pierre Jolivet <pierre at joliv.et>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 13 Apr 2022, at 3:30 PM, Zhuo Chen <chenzhuotj at gmail.com> wrote:
>>>>>>>>
>>>>>>>> Dear Matthew and Mark,
>>>>>>>>
>>>>>>>> Thank you very much for the reply! Much appreciated!
>>>>>>>>
>>>>>>>> The question was about a 1D problem. I think I should say core 1
>>>>>>>> has row 1:32 instead of 1:32, 1:32 as it might be confusing.
>>>>>>>>
>>>>>>>> So the overlap is extended to both directions for the middle
>>>>>>>> processor but only toward the increasing direction for the first processor
>>>>>>>> and the decreasing direction for the last processor. In 1D, this makes
>>>>>>>> sense as the domain is contiguous. However, in 2D with domain decomposition
>>>>>>>> with spacial overlaps, this overlapping would not work as one subdomain can
>>>>>>>> have several neighbor domains. Mark mentioned generalized ASM, is that the
>>>>>>>> correct direction that I should look for?
>>>>>>>>
>>>>>>>>
>>>>>>>> What is it that you want to do exactly?
>>>>>>>> If you are using a standard discretisation kernel, e.g., piecewise
>>>>>>>> linear finite elements, MatIncreaseOverlap() called by PCASM will generate
>>>>>>>> an overlap algebraically which is equivalent to the overlap you would have
>>>>>>>> gotten geometrically.
>>>>>>>> If you know that “geometric” overlap (or want to use a custom
>>>>>>>> definition of overlap), you could use
>>>>>>>> https://petsc.org/release/docs/manualpages/PC/PCASMSetLocalSubdomains.html
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Pierre
>>>>>>>>
>>>>>>>> Best regards.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 13, 2022 at 9:14 PM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Apr 13, 2022 at 9:11 AM Mark Adams <mfadams at lbl.gov>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 13, 2022 at 8:56 AM Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 13, 2022 at 6:42 AM Mark Adams <mfadams at lbl.gov>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> No, without overlap you have, let say:
>>>>>>>>>>>> core 1:   1:32, 1:32
>>>>>>>>>>>> core 2:   33:64,  33:64
>>>>>>>>>>>>
>>>>>>>>>>>> Overlap will increase the size of each domain so you get:
>>>>>>>>>>>> core 1:   1:33, 1:33
>>>>>>>>>>>> core 2:   32:65,  32:65
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I do not think this is correct. Here is the algorithm. Imagine
>>>>>>>>>>> the matrix is a large graph. When you divide rows, you
>>>>>>>>>>> can think of that as dividing the vertices into sets. If overlap
>>>>>>>>>>> = 1, it means start with my vertex set, and add all vertices
>>>>>>>>>>> that are just 1 edge away from my set.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think that is what was said. You increase each subdomain by one
>>>>>>>>>> row of vertices.
>>>>>>>>>> So in 1D, vertex 32 and 33 are in both subdomains and you have an
>>>>>>>>>> overlap region of size 2.
>>>>>>>>>> They want an overlap region of size 1, vertex 33.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is true, but I did not think they specified a 1D mesh.
>>>>>>>>>
>>>>>>>>>   Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> What you want is reasonable but requires PETSc to pick a
>>>>>>>>>>>> separator set, which is not well defined.
>>>>>>>>>>>> You need to build that yourself with gasm (I think) if you want
>>>>>>>>>>>> this.
>>>>>>>>>>>>
>>>>>>>>>>>> Mark
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 13, 2022 at 3:17 AM Zhuo Chen <chenzhuotj at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I hope that everything is going well with everybody.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a question about the PCASMSetOverlap. If I have a
>>>>>>>>>>>>> 128x128 matrix and I use 4 cores with overlap=1. Does it mean that from
>>>>>>>>>>>>> core 1 to core 4, the block ranges are (starting from 1):
>>>>>>>>>>>>>
>>>>>>>>>>>>> core 1:   1:33, 1:33
>>>>>>>>>>>>> core 2:   33:65,  33:65
>>>>>>>>>>>>> core 3:   65:97,  65:97
>>>>>>>>>>>>> core 4:   95:128, 95:128
>>>>>>>>>>>>>
>>>>>>>>>>>>> Or is it something else? I cannot tell from the manual.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Many thanks in advance.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Zhuo Chen
>>>>>>>>>>>>> Department of Astronomy
>>>>>>>>>>>>> Tsinghua University
>>>>>>>>>>>>> Beijing, China 100084
>>>>>>>>>>>>> *https://czlovemath123.github.io/
>>>>>>>>>>>>> <https://czlovemath123.github.io/>*
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Zhuo Chen
>>>>>>>> Department of Astronomy
>>>>>>>> Tsinghua University
>>>>>>>> Beijing, China 100084
>>>>>>>> *https://czlovemath123.github.io/
>>>>>>>> <https://czlovemath123.github.io/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Zhuo Chen
>>>>>>> Department of Astronomy
>>>>>>> Tsinghua University
>>>>>>> Beijing, China 100084
>>>>>>> *https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhuo Chen
>>>>> Department of Astronomy
>>>>> Tsinghua University
>>>>> Beijing, China 100084
>>>>> *https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Zhuo Chen
>>>> Department of Astronomy
>>>> Tsinghua University
>>>> Beijing, China 100084
>>>> *https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
>>>>
>>>
>>>
>>> --
>>> Zhuo Chen
>>> Department of Astronomy
>>> Tsinghua University
>>> Beijing, China 100084
>>> *https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
>>>
>>>
>>>
>>
>> --
>> Zhuo Chen
>> Department of Astronomy
>> Tsinghua University
>> Beijing, China 100084
>> *https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Zhuo Chen
Department of Astronomy
Tsinghua University
Beijing, China 100084
*https://czlovemath123.github.io/ <https://czlovemath123.github.io/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220414/606878f3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dd.png
Type: image/png
Size: 42088 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220414/606878f3/attachment-0001.png>

More information about the petsc-users mailing list