[petsc-users] MatMatSolve in sequential call

[Jack Poulson]

On Tue, Aug 21, 2012 at 6:24 PM, Jack Poulson <jack.poulson at gmail.com>wrote:

> On Tue, Aug 21, 2012 at 6:22 PM, Jack Poulson <jack.poulson at gmail.com>wrote:
>
>> On Tue, Aug 21, 2012 at 3:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Tue, Aug 21, 2012 at 11:47 AM, Jack Poulson <jack.poulson at gmail.com>wrote:
>>>
>>>> On Tue, Aug 21, 2012 at 11:42 AM, Alexander Grayver <
>>>> agrayver at gfz-potsdam.de> wrote:
>>>>
>>>>>  On 21.08.2012 18:32, Matthew Knepley wrote:
>>>>>
>>>>>   MUMPS takes only several minutes and 6 GB of memory to factorize
>>>>>> it.
>>>>>> This factorization gives residual on the order of 10e-12 and solution
>>>>>> is indeed correct.
>>>>>>
>>>>>> Nevertheless, you're right, there is numerical null-space in this
>>>>>> matrix since it comes
>>>>>> from the discretization of equation that contains curl curl operator,
>>>>>> but practically this
>>>>>> case is not really the worst one.
>>>>>>
>>>>>
>>>>>  This makes no sense whatsoever. How can you LU factor a matrix that
>>>>> has a null space?
>>>>>
>>>>>
>>>>> Matt,
>>>>>
>>>>> I'm not sure that I correctly used term numerical null-space in my
>>>>> post.
>>>>> The equation is
>>>>>
>>>>> curl curl E + kE = -J,
>>>>>
>>>>> where k is a function of frequency and conductivity, whenever one of
>>>>> them becomes small this term gets vanishingly small thus we have problems
>>>>> since curl curl operator has nontrivial null-space by definition. So let's
>>>>> say solving this equation for low frequencies and for models containing air
>>>>> is difficult.
>>>>>
>>>>> What kind of magic is inside MUMPS I don't know, but it is able to
>>>>> handle such cases (e.g. SuperLU and PaStiX fail).
>>>>>
>>>>> Also, if it matters, I'm talking about LDLt factorization in MUMPS.
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Alexander
>>>>>
>>>>>  You can find Vasseur's talk on this exact subject here:
>>>> http://graal.ens-lyon.fr/MUMPS/doc/ud_2010/Vasseur_talk.pdf
>>>
>>>
>>> I was wrong, this is not nonsense. However, for curl curl the null space
>>> grows with matrix dimension, and
>>> as far as I can tell from the slides, the null space determination is
>>> not scalable (Jack correct me if I am wrong).
>>> Also, they gave no timings, so I suspect null space determination is
>>> slow.
>>>
>>> I don't think any other LU we have will do this, so if you have null
>>> spaces you are stuck with MUMPS.
>>>
>>>     Matt
>>>
>>>
>>
>> It's not something that I've studied in detail, but I believe that it
>> isn't that the behavior will not be that different from "difficult"
>> nonsingular cases (i.e., where a large number of pivots do not satisfy the
>> thresholding condition and must be delayed to the parent front). If the
>> null space is large, then I would expect this to impact performance
>> significantly. I would expect it to make load balancing much more
>> difficult. In practice, this might lead to nonscalability, as it is
>> sophisticated algorithm.
>>
>> Jack
>>
>
> Please ignore the atrocity that was my attempt at a first sentence in the
> previous email: the point is that the delayed pivot mechanism is also used
> within the standard threshold pivoted LU factorization.
>
> Jack
>

Hmmm, I just noticed that slide 25 says that (at least in 2010) the root
node is processed sequentially in order to handle nontrivial nullspaces. I
would contact the developers, but the description of ICNTL(24) on pg. 29 of
the most recent manual (http://mumps.enseeiht.fr/doc/userguide_4.10.0.pdf)
makes it seem that the root node must still be factored sequentially if an
exact zero pivot is detected.

This will certainly limit the parallelism, as the computational complexity
of factoring the root node is usually essentially the same as factoring the
entire problem.

Jack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120821/32f88513/attachment.html>