[petsc-users] Fwd: How PETSc solves Ax=b in parallel

[paul zhang]

That is a good one. I mean for my case. which method can I try?


On Tue, Oct 22, 2013 at 4:14 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 22, 2013 at 3:12 PM, paul zhang <paulhuaizhang at gmail.com>wrote:
>
>> One more question, can I solve the system in parallel?
>>
>
> Yes, or you would be using ETSC :)
>
>    Matt
>
>
>>
>> On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com>wrote:
>>>
>>>> Thanks for the answer. It makes sense.
>>>>
>>>> However, in my case, matrix A is huge and rather sparse, which also
>>>> owns a pretty good diagonal structure although there are some other
>>>> elements are nonzero. I have to  look for a better way to solve the system
>>>> more efficiently. If in parallel, it is even better.
>>>>
>>>> Attached is an example for A's structure. The pink block is a matrix
>>>> with 10x10 elements. The row or column in my case can be in million size.
>>>>
>>>
>>> The analytic character of the operator is usually more important than
>>> the sparsity structure for scalable solvers.
>>> The pattern matters a lot for direct solvers, and you should definitely
>>> try them (SuperLU_dist or MUMPS in PETSc).
>>> If they use too much memory or are too slow, then you need to
>>> investigate good preconditioners for iterative methods.
>>>
>>>    Matt
>>>
>>>
>>>> Thanks again.
>>>> Paul
>>>>
>>>>
>>>>
>>>>  --
>>>> Huaibao (Paul) Zhang
>>>> *Gas Surface Interactions Lab*
>>>> Department of Mechanical Engineering
>>>> University of Kentucky,
>>>> Lexington, KY, 40506-0503
>>>> *Office*: 216 Ralph G. Anderson Building
>>>> *Web*:gsil.engineering.uky.edu
>>>>
>>>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>>>
>>>>> Hi Jed,
>>>>>
>>>>> Thanks a lot for your answer. It really helps. I built parts of the
>>>>> matrix on each processor, then collected them into a global one according
>>>>> to their global position. Actually I used two MPI function instead of the
>>>>> one in the example, where the local size, as well as the global size is
>>>>> given.
>>>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>>>
>>>>> My continuing question is since the iteration for the system is
>>>>> global. Is it more efficient if I solve locally instead. ie. solve parts on
>>>>> each of the processor instead of doing globally.
>>>>>
>>>>
>>>> No, because this ignores the coupling between domains.
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> Thanks again,
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>
>>>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>>>
>>>>>> > I am using KSP, more specifically FGMRES method, with MPI to solve
>>>>>> Ax=b
>>>>>> > system. Here is what I am doing. I cut my computation domain into
>>>>>> many
>>>>>> > pieces, in each of them I compute independently by solving fluid
>>>>>> equations.
>>>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>>>> > information and load it to a whole A matrix.
>>>>>>
>>>>>> I hope you build parts of this matrix on each processor, as is done in
>>>>>> the examples.  Note the range Istart to Iend here:
>>>>>>
>>>>>>
>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>>>
>>>>>> > My question is how PETSc functions work in parallel in my case.
>>>>>> There are
>>>>>> > two guesses to me. First, PETSc solves its own matrix for each
>>>>>> domain using
>>>>>> > local processor, although A is a global. For the values like number
>>>>>> of
>>>>>> > iterations, solution vector, their numbers should have equaled to
>>>>>> the
>>>>>> > number of processors I applied, but I get only one value for each
>>>>>> of them.
>>>>>> > The reason is that the processors must talk with each other once
>>>>>> all of
>>>>>> > their work is done, that is why I received the "all reduced" value.
>>>>>> This is
>>>>>> > more logical than my second guess.
>>>>>>
>>>>>> It does not work because the solution operators are global, so to
>>>>>> solve
>>>>>> the problem, the iteration must be global.
>>>>>>
>>>>>> > In the second one, the system is solved in parallel too. But PETSc
>>>>>> function
>>>>>> > redistributes the global sparse matrix A to each of the processors
>>>>>> after
>>>>>> > its load is complete. That is to say now each processor may not
>>>>>> solve the
>>>>>> > its own partition matrix.
>>>>>>
>>>>>> Hopefully you build the matrix already-distributed.  The default
>>>>>> _preconditioner_ is local, but the iteration is global.  PETSc does
>>>>>> not
>>>>>> "redistribute" the matrix automatically, though if you call
>>>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>>>> choose them.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Huaibao (Paul) Zhang
>>>>> *Gas Surface Interactions Lab*
>>>>>  Department of Mechanical Engineering
>>>>> University of Kentucky,
>>>>> Lexington,
>>>>> KY, 40506-0503*
>>>>> Office*: 216 Ralph G. Anderson Building
>>>>> *Web*:gsil.engineering.uky.edu
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Huaibao (Paul) Zhang
>> *Gas Surface Interactions Lab*
>> Department of Mechanical Engineering
>> University of Kentucky,
>> Lexington,
>> KY, 40506-0503*
>> Office*: 216 Ralph G. Anderson Building
>> *Web*:gsil.engineering.uky.edu
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/5ca7fd58/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/5ca7fd58/attachment-0001.png>