[petsc-users] PCGASMSetLocalSubdomains

Dmitry Karpeev karpeev at mcs.anl.gov
Tue May 15 11:06:53 CDT 2012 

On Tue, May 15, 2012 at 11:05 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

>
> On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:
>
>
>
> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> thanks for reply!
>> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
>>
>>
>>
>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Dmitry,
>>>
>>> thanks for remind.  I have a new question about PCASM / PCGASM:
>>>
>>> can I get the restricted extension operators, which maps an overlapping
>>> subdomain solution
>>> to the global domain?
>>>
>>
>> I'm not exactly sure what you mean.
>>
>> Are you talking about embedding the subdomain vectors back into the
>> original vector?
>>
>>
>> Yes, exactly.
>>
>> If so, there is substantial difference in how this is handled in ASM and
>> GASM:
>> ASM has a bunch of sequential vectors that can be scattered back into the
>> global vector,
>>
>>
>> Yes. Is there a method to get the scatter?
>>
>
> In the ASM case it's a bunch of scatters -- one for each subdomain.
>  Currently there is no method to
> retrieve them.
>
>
> this hint is very helpful. Thanks!
>
> What requires this functionality?
>
>
> I am writing some modified ASM method. In construction of energy
> minimization coarse basis,
> I need to solve individual subdomain problems and not to sum them, just to
> extend them separately.
> I wonder whether you guys have ever done this coarse basis.
>
Is there a reference where the basis is described?
Dmtiry.

>
> Thanks,
> Hui
>
>
> In the ASM case you can construct the scatters yourself easily enough,
> since you have all of the requisite information -- the array of subdomain
> ISs and the global vector x.
> The only piece of data you might not have is the set of outer subdomains
> that have been obtained
> by applying overlap increase to the original inner (nonoverlapping)
> subdomains.
>
>>
>> because the subdomains are always local to at most one processor.
>>
>> In the GASM case this is rather different, since the subdomains can live
>> on arbitrary subcommunicators
>> and there is only one scatter, which is applied to the direct sum of all
>> the subdomain vectors on the original communicator. I'm not sure how useful
>> that last scatter would be for you, since the details of the structure
>> of the direct sum vector are internal to GASM.
>>
>>
>> I would prefer to have the scatter for individual subdomain before direct
>> sum.
>> But if I can get the scatter PCGASM has, maybe it is still useful. Please
>> tell me how to get it?
>>
> There are no individual subdomain scatters, but, as in the case of ASM,
> you can  construct them
> easily enough, except that those would have to operate on subcommunicators.
> In GASM we pack them into a single scatter on the original communicator.
> Currently there is no method
> to expose this scatter. Why do you need this functionality?
>
> Dmitry.
>
>> Thanks!
>>
>>
>> Dmitry.
>>
>>>
>>> Thanks!
>>>
>>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>>>
>>> There are some additional minor fixes that mostly have to do with
>>> outputting the subdomain information with  -pc_gasm_view_subdomains (in
>>> PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>> You might want to pull those latest patches, but it won't interfere with
>>> your work if you don't use subdomain output.
>>>
>>> Thanks.
>>> Dmitry.
>>>
>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Dmitry,
>>>>
>>>> thanks for reply. I re-download the codes and tried it again and now it
>>>> works correctly!
>>>>
>>>> Everything seems ok.
>>>>
>>>> Thanks,
>>>> Hui
>>>>
>>>>
>>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>>>
>>>> Hui,
>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>> One thing that looks odd is that the output below claims the PC is of
>>>> type "asm", even though you are running with -dd_type gasm.  Could you
>>>> verify that's the correct output?
>>>>
>>>> Here's the output I get with
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>>>> asm -dd_ksp_view
>>>>
>>>> PC Object:(dd_) 1 MPI processes
>>>>    type: asm
>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>     Local solve is same for all blocks, in the following KSP and PC
>>>> objects:
>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>       type: preonly
>>>>       maximum iterations=10000, initial guess is zero
>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>       left preconditioning
>>>>       using NONE norm type for convergence test
>>>>       <snip>
>>>>
>>>> and with
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>>>> gasm -dd_ksp_view
>>>>
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: gasm
>>>>   Generalized additive Schwarz:
>>>>   Restriction/interpolation type: RESTRICT
>>>>   user-defined overlap
>>>>   total number of subdomains = 2
>>>>   number of local subdomains = 2
>>>>   max number of local subdomains = 2
>>>>   [0:1] number of locally-supported subdomains = 2
>>>>   Subdomain solver info is as follows:
>>>>   <snip>
>>>>
>>>> What convergence are you seeing with the two PC types?  It should be
>>>> the same with 1 and 2 procs for both PCASM and PCGASM.
>>>>
>>>> Thanks.
>>>> Dmitry.
>>>>
>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>>
>>>>> Dmitry,
>>>>>
>>>>> I got the newest petsc-dev and I run the test by
>>>>>
>>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>>>
>>>>> which gives the following output
>>>>>
>>>>> PC Object:(dd_) 1 MPI processes
>>>>>   type: asm
>>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>>                                               ^^^
>>>>>                                          note the above number, it
>>>>> should be 2
>>>>>
>>>>> While PCASM has no such problem.
>>>>>
>>>>> Thanks,
>>>>> Hui
>>>>>
>>>>>
>>>>>
>>>>> Hui,
>>>>>
>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some
>>>>> upcoming API changes)
>>>>> that should verify that.
>>>>>
>>>>> Let me know if it works for you.
>>>>> Thanks.
>>>>> Dmitry.
>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> thanks for useful hints.  Good day!
>>>>>>
>>>>>> Hui
>>>>>>
>>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy
>>>>>> and recreate the matrices (including calling
>>>>>> your modification subroutine), but not the subdomains or the
>>>>>> subdomain solvers.
>>>>>> If you just want to modify the submatrices, you can call
>>>>>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>> (in the same order as the subdomains were set). That's a bit of a
>>>>>> hack, since you will essentially be modifying the PC's internal data
>>>>>> structures.  As long as you are careful, you should be okay, since you
>>>>>> already effectively have the same type of access to the submatrices through
>>>>>> the Modify callback.
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one
>>>>>>> time.
>>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>>>>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>>>>>> the same routine for modifying but with
>>>>>>> different user context for the modifying routine.
>>>>>>>
>>>>>>> What can I do for this task?  Currently, I destroy the KSP and
>>>>>>> re-construct it. I guess
>>>>>>> even for PCASM I can re-use it because the partition of subdomains
>>>>>>> remain the same.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Hui,
>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>> Dmitry.
>>>>>>>
>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dmitry,
>>>>>>>>
>>>>>>>> is there any news about PCGASM?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>>>
>>>>>>>> Okay, thanks.
>>>>>>>> I'll take a look.
>>>>>>>>
>>>>>>>> Dmitry.
>>>>>>>>
>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <
>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>
>>>>>>>>> For reference, my results are attached.
>>>>>>>>>
>>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>>>>>> different from others)
>>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thank you,
>>>>>>>>> Hui
>>>>>>>>>
>>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I have a new problem: the results from ASM and GASM are
>>>>>>>>>>> different and it seems
>>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical
>>>>>>>>>>> tests are with
>>>>>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>>>>>> problems when
>>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices,
>>>>>>>>>>> there are problems
>>>>>>>>>>> with GASM but no problems with ASM.
>>>>>>>>>>>
>>>>>>>>>>> For example, I use two subdomains. In the first case each
>>>>>>>>>>> subdomain is supported by
>>>>>>>>>>> one processor and there seems no problem with GASM. But when I
>>>>>>>>>>> use run my program
>>>>>>>>>>> with only one proc. so that it supports both of the two
>>>>>>>>>>> subdomains, the iteration
>>>>>>>>>>> number is different from the first case and is much larger.  On
>>>>>>>>>>> the other hand
>>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are the solutions the same?
>>>>>>>>>> What problem are you solving?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>>>>>> results with one or
>>>>>>>>>> two processors. But GASM did not.
>>>>>>>>>>
>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions
>>>>>>>>> in the case of two domains per processor?
>>>>>>>>>
>>>>>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>>>
>>>>>>>>> That would be helpful.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Dmitry.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dmitry.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>
>>>>>>>>>>> You should be able to.
>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>> I am in transit right now, but I can take a closer look in
>>>>>>>>>>> Friday.
>>>>>>>>>>>
>>>>>>>>>>> Dmitry
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>
>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>>>>>> another question
>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has
>>>>>>>>>>> the prototype
>>>>>>>>>>>
>>>>>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void
>>>>>>>>>>> *ctx);
>>>>>>>>>>>
>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the
>>>>>>>>>>> same as the rows
>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains()
>>>>>>>>>>> only accepts
>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood
>>>>>>>>>>> something?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> As I tested, the row and col are always the same.
>>>>>>>>>>>
>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping()
>>>>>>>>>>> for the submat's
>>>>>>>>>>> in the above func()?
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>
>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>> There is no good way to help the user assemble the subdomains at
>>>>>>>>>>> the moment beyond the 2D stuff.
>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>
>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a
>>>>>>>>>>> "coloring" of indices to an array of ISs,
>>>>>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>>>>>> supports that color. It is
>>>>>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>>>>>> feedback on any problems you encounter.
>>>>>>>>>>>
>>>>>>>>>>> Dmitry.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>>>>>> supported by
>>>>>>>>>>>> multiple processors, shall I always create the arguments
>>>>>>>>>>>> 'is[s]' and 'is_local[s]'
>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>>>>>> subdomain 's'?
>>>>>>>>>>>>
>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> <gasm_test.c>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/65dae8c7/attachment-0001.htm>

More information about the petsc-users mailing list