[petsc-users] PCGASMSetLocalSubdomains

Mon May 21 14:32:25 CDT 2012

Okay, thanks.
I'll look into it.
Dmitry.

On Mon, May 21, 2012 at 2:11 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the
> output becomes the
> same as PCASM. See attached.
>
>
>
>
> On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
>
> Dmitry,
>
> finally, I am able to make a simple program and reproduce the error. Since
> I had moved to
> petsc-dev (before I used petsc-3.2), the error is a little different.
>
> Download the program attached and run the run.sh.  You will see the output
> from PCASM.
> There is a submatrix (submat[1] on rank 0) which has a row like
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1)
>
> Note that the column 2 is placed in the last and the column numbers are
> not in the normal order.
> Following that, we can also see the error message like
>
>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
>
> You can also try with GASM, use
>
>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
>
> which partitions the domain to 1 * 2 and mapped to the two processors, and
> for each processor
> we further partition its local domain to 2 * 1 subdomains.
>
> Then, you will see that the row 1 of submat[1] on rank 1 becomes
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1)
>
> which is totally wrong because from the IS for this overlapping subdomain
> and the big matrix A,
> row 1 should have non-zeros like the output from PCASM.
>
> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and
> something goes wrong
> in asm.c and gasm.c.
>
> Thanks!
> Hui
>
>
> <GASM_test.zip>
>
>
>
>
>
> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> I have the following MatView output:
>>
>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4,
>> -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793)
>>
>> Note that there are two values for the column 1, is this normal?
>>
>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and
>> KSPGetOperators(subksp[0], ...).
>>
>
> This is odd. Can you explain a bit more what leads up to this, so we can
> try and reproduce the problem?
> Thanks.
> Dmitry.
>
>>
>> Thanks!
>>
>>
>>
>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
>> recreate the matrices (including calling
>> your modification subroutine), but not the subdomains or the subdomain
>> solvers.
>> If you just want to modify the submatrices, you can call
>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a
>> hack, since you will essentially be modifying the PC's internal data
>> structures.  As long as you are careful, you should be okay, since you
>> already effectively have the same type of access to the submatrices through
>> the Modify callback.
>>
>> Dmitry.
>>
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> I just have a question about reuse of PCASM or PCGASM.
>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>> the same routine for modifying but with
>>> different user context for the modifying routine.
>>>
>>> What can I do for this task?  Currently, I destroy the KSP and
>>> re-construct it. I guess
>>> even for PCASM I can re-use it because the partition of subdomains
>>> remain the same.
>>>
>>> Thanks!
>>>
>>>
>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>>
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> is there any news about PCGASM?
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>
>>>> Okay, thanks.
>>>> I'll take a look.
>>>>
>>>> Dmitry.
>>>>
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>> For reference, my results are attached.
>>>>>
>>>>> asm1.txt for asm with 1 process,
>>>>> asm2.txt for asm with 2 processes,
>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>> different from others)
>>>>> gasm2.txt for gasm with 2 processes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> thank you,
>>>>> Hui
>>>>>
>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I have a new problem: the results from ASM and GASM are different
>>>>>>> and it seems
>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>>> are with
>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>> problems when
>>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>>> are problems
>>>>>>> with GASM but no problems with ASM.
>>>>>>>
>>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>>> is supported by
>>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>>> run my program
>>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>>> the iteration
>>>>>>> number is different from the first case and is much larger.  On the
>>>>>>> other hand
>>>>>>> ASM has no such problem.
>>>>>>>
>>>>>>
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>>>
>>>>>>
>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>> results with one or
>>>>>> two processors. But GASM did not.
>>>>>>
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in
>>>>> the case of two domains per processor?
>>>>>
>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>> I can prepare a simpler example to show this difference.
>>>>>>
>>>>> That would be helpful.
>>>>> Thanks.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> You should be able to.
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>
>>>>>>> Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>
>>>>>>> Hi Dmitry,
>>>>>>>
>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>> another question
>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>>> prototype
>>>>>>>
>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>
>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>> the rows
>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>> accepts
>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>
>>>>>>>
>>>>>>> As I tested, the row and col are always the same.
>>>>>>>
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>>> the submat's
>>>>>>> in the above func()?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Yes, that's right.
>>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>>> moment beyond the 2D stuff.
>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>
>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>>> of indices to an array of ISs,
>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>> supports that color. It is
>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>> feedback on any problems you encounter.
>>>>>>>
>>>>>>> Dmitry.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>> supported by
>>>>>>>> multiple processors, shall I always create the arguments 'is[s]'
>>>>>>>> and 'is_local[s]'
>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>> subdomain 's'?
>>>>>>>>
>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/7b569045/attachment-0001.html>